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ABSTRACT 

This report provides an overview of the findings of 3 studies 
conducted between 1995 and 1998 on school-based performance award (SBPA) 
programs. Such programs provide teachers, and often other school, staff, with 
pay bonuses when their school as a whole achieves specific educational 
objectives. The studies focused on programs in the state of Kentucky, the 
state of Maryland, and the Charlotte-Mecklenburg district in North Carolina. 
Kentucky and Charlotte-Mecklenburg paid salary bonuses for improvement in 
student outcomes toward a specific goal to all teachers in a school. Maryland 
provided awards to schools as school-improvement funds, not directly to 
teachers as bonuses. The report concentrates on three topics: teachers* 
motivational reactions to the programs; the degree to which teachers* 
motivational reactions are related to school performance; and principals' 
assessments of the effects of the programs. Data were gathered from 
interviews and surveys of teachers and principals. The results suggest that 
SBPA programs are a potentially useful tool for improving student success. 
Their full potential may, however, be difficult to reach. The programs 
studied appeared to have a focusing effect on teaching practices (both 
positive and negative effects) , collaboration and teamwork, and resource 
alignment. (Contains 16 tables, 2 figures, and 68 references.) (WFA) 
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INTRODUCTION 



B uilding on a strong foundation of research and experience with educational reform 
across the nation, in the 1990s educational policymakers undertook sophisticated 
comprehensive educational reform efforts. The centerpiece of these reforms was the 
creation of state educational standards, assessments, and goals, and the realignment of 
state and local resources to support the achievement of these goals. A number of states and dis- 
tricts have embraced the idea of performance-based accountability (Fuhrman, 1 999), in which 
rewards and sometimes sanctions are used as incentives for measurable improvement in student 
achievement. Some jurisdictions have modified their teacher compensation systems in order to 
provide incentives that support improving student achievement. Learning from the relative fail- 
ure of prior efforts to realign teacher compensation via individual performance pay systems 
(Mumane and Cohen, 1986), states and districts undertaking compensation reform in the 1990s 
have tended to focus on innovations such as school-based performance awards (SBPAs) and pay 
based on knowledge and skills, which are thought to be better ways of supporting collaborative 
cultures in schools (Odden and Kelley, 1997). This report provides an overview of the findings 
of a series of studies of SBPA programs conducted by the Consortium for Policy Research in 
Education between 1995 and 1998. The research reported here focuses on two such programs, in 
Kentucky and Charlotte-Mecklenburg (North Carolina), that pay teachers bonuses, and on one 
program, in Maryland, that provides the award to schools to use for activities and improvements. 



WHAT ARE SCHOOL-BASED 
PERFORMANCE AWARD PROGRAMS? 



School-based performance award programs provide teachers and often other school staff with 
pay bonuses when their school as a whole achieves specific educational objectives. These objec- 
tives typically involve improvement in student achievement as measured by some form of test. 
These programs differ from merit pay in that all teachers in a school receive the bonus based on 
meeting an objective, a predefined schoolwide goal, rather than based on individual teacher per- 
formance as rated by administrators. 

In addition to pay bonuses, SBPA programs might also include money for school improvements, 
public recognition, and exemption from regulatory requirements. These award programs are 
meant to provide an incentive for teachers to work toward improved student performance, while 
avoiding the divisiveness and discomfort that can accompany merit pay. The group nature of the 
goal is intended to encourage teachers to work together, and the collective nature of the incentive 
may provide an important symbolic focus on key educational outcomes (Odden and Kelley, 
1997). During the 1990s, the states of Kentucky and North Carolina, and the Dallas, Philadel- 
phia, and Charlotte-Mecklenburg school districts had programs that paid bonuses to teachers. 
Programs that provided money to schools were implemented in Maryland, Pennsylvania, South 
Carolina, and in the districts of Boston and Miimeapolis. To some extent, the interest in school- 
based performance award programs in the education sector parallels the rising use of group per- 
formance or incentive awards in the U.S. private sector. There, the reported results of group re- 
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wards has generally been positive (Heneman and Judge, in press; McAdams and Hawk, 1994). 
The experience in the educational sector is still to be assessed, and the research reported here is 
intended as a contribution to that assessment. 

Understanding the effects of school-based performance award programs on school and teacher 
performance is particularly important in light of past research on individual performance (or 
“merit”) pay plans for teachers (Hatry, Greiner, and Ashford, 1994; Jacobson, 1987; Johnson, 
1986; Mumane and Cohen, 1986). This literature has consistently shown that merit pay plans 
tend to be ineffective and short-lived. The programs create competition among teachers by iden- 
tifying a small proportion of the “best” teachers, which is inconsistent with the findings of other 
research that suggests that the “best” schools require high levels of professional collaboration 
among teachers (for example, Rosenholtz, 1989). In addition, the definition of “best” is rarely 
specified, and teachers often perceive that recognition is unfairly distributed among teachers. 
Finally, there is inconsistent support for merit pay plans over time. The programs that remain in 
place tend to operate in districts with high pay for teachers, and they evolve to include more and 
more teachers, gradually linking the additional pay to additional job responsibilities (Mumane 
and Cohen, 1986; Odden and Kelley, 1997). The poor performance of merit pay plans in educa- 
tion is similar to the experience in the private sector, particularly in organizations where success 
requires collaboration and teamwork (Lawler, 1990). 

This report concentrates on three main topics: teachers’ motivational reactions to these programs, 
the degree to which teachers’ motivational reactions are related to school performance (as de- 
fined by each program), and principals’ assessments of the effects of the programs. It should be 
noted that our research does not tell us whether or not school-based performance award programs 
“cause” improvements in student achievement and, if so, by how much. This would require the 
use of an experimental or quasi-experimental research design, which was not possible given the 
need to study existing implementation sites and the absence of equivalent or nearly equivalent 
states or districts to serve as comparisons. Instead, we used a combination of on-site interviews 
and survey questionnaires to assess teacher and principal attitudinal responses to these programs 
and to determine whether the programs created conditions likely to motivate teachers to work 
toward improving student achievement. We also aggregated these responses to the school level 
in order to examine whether Kentucky and Charlotte-Mecklenburg schools, where teachers 
would be expected to be more motivated on the average, were more successful in improving stu- 
dent achievement. 



AN OVERVIEW OF THE PROGRAMS WE 
STUDIED 



Kentucky’s Accountability Program 

The Kentucky Accountability Program held schools accountable for improvement in school per- 
formance in seven academic subjects (reading, writing, math, science, social studies, 
arts/humanities, and vocational/practical living) and school-level indicators such as student at- 
tendance, retention, and dropout rates, and successful transition to adult life (see Kelley, 1998b; 
Kelley and Protsik, 1997). The school performance index was called KIRIS, for Kentucky In- 
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structional Results Information System. Both the accountability program and the assessment in- 
strument were created in 1990 as part of Kentucky’s state education reforms. The assessment 
instrument, which consisted of portfolio entries, performance events, open-response, and multi- 
ple choice questions, was administered in grades four and five, seven and eight, and eleven and 
twelve. Student scores were placed in one of four categories on the assessment: novice, appren- 
tice, proficient, and distinguished. The calculation of the academic portion of the KIRIS index 
was based on the proportion of the school’s students who score in each category multiplied by a 
weight that gives more credit for students who performed at the higher levels. 

Beginning in 1991-92, the state set a series of two-year goals for each school. Schools were ex- 
pected to increase their KIRIS scores 10 percent of the distance between their school’s baseline 
score (initially the 1 99 1 -92 score, then the score from the prior cycle) and a long-term target 
(equivalent to 100 percent of students scoring at the proficient level). If schools exceeded their 
goal, they were designated as “reward” schools and received a pool of award funds that could be 
used for any purpose, including salary bonuses, as decided by a vote of the school’s certified 
staff. The amount awarded to each school was based on the degree to which the school exceeded 
its goals and the number of certified staff. Fifty-two award amount levels were established. 

State law set the maximum award amount at approximately 10 percent of the average teacher 
salary in the five Kentucky districts having the highest average salary ($3,690 for Cycle 1). The 
minimum amount was set at 50 percent of this maximum. Certified staff at each winning school 
decided how to use the reward, including whether only teachers, or teachers and support staff, 
would be paid bonuses. In the first accountability cycle, 39 percent of schools exceeded their 
performance target and received performance awards, and teachers in over 98 percent of award 
schools voted to use all or part of the money for salary bonuses. In the second cycle, 40 percent 
of schools achieved reward status, and 88 percent of the schools voted to use all or part of the 
money for salary bonuses (Gong, 1997; Kentucky Department of Education, 1996; Reidy, 1995). 
The average bonus paid to teachers at the end of Cycle 2 (1995-96) was approximately $2,600. 

Schools that met their target were termed “successful” and received neither bonuses nor sanc- 
tions. Schools that improved over their baseline KIRIS score but failed to reach the target were 
called “improving.” They had to submit a transformation plan to the state that described how 
they planned to meet their achievement goals in the future. Schools that dropped below baseline 
were designated “in decline” and were assigned a distinguished educator, received improvement 
funds, and had to submit a transformation plan. Distinguished educators were experienced teach- 
ers or administrators from the state of Kentucky, hired and trained by the state to provide techni- 
cal assistance to schools to help them meet accountability goals. Schools that dropped more than 
five points below baseline were designated “in crisis.” They were treated like schools in decline, 
with one notable exception. The distinguished educators assigned to crisis schools had broad 
powers to terminate teachers and override school site council decisions if they deemed it appro- 
priate (Kentucky Department of Education, 1995). In the first biennium of the program, the crisis 
category was suspended due to concerns about the reliability of the assessment instrument. The 
crisis category was reinstituted for the second biennium, and nine schools were designated “in 
crisis” (Kentucky Department of Education, 1995; Kentucky Department of Education, 1996). 

Our research, conducted from 1995 through 1998, concerns the program as described above. 

Due to a variety of factors, including difficulties with the assessment portion of the program 
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(Blackford, 1997; Lawton, 1997; Strecklow, 1997), the state modified the program in 1998 to 
change the way school goals are set, to change the nature of the sanctions, and to eliminate the 
possibility of paying bonuses to individual staff members (Cody, 1998). 

Charlotte-Mecklenburg’s Benchmark Goals Program 

Under the direction of a new superintendent, the Charlotte-Mecklenburg School District estab- 
lished its Benchmark Goals Program in 1991 in an attempt to reverse a record of low student 
achievement and limited success with minority students (see Heneman, 1998). At the time, the 
district had about 90,000 students (60 percent white and 40 percent African-American or other) 
and was growing at a rate of 3,500 students per year. This growth rate and racial composition 
continue to the present. Approximately 33 percent of the schools in the district are magnet 
schools. Teacher salaries in North Carolina were comparatively low, with an average of $30,564 
in 1995-96, ranking 42nd among the states. A statewide teacher salary schedule set minimums 
for all districts, though district supplements were permitted. The district provided a small sup- 
plement, placing its starting teachers at $23,600, which was $2,300 above the state minimum 
(Cenziper, 1997). 

The Benchmark Goal Program became operational in 1992-93, with annual implementation and 
management plans extending through the 1996-97 school year. In 1996-97, the program entered 
a transitional period due to the hiring of a new superintendent and the beginning of a new state- 
wide accountability program for school districts called the ABC’s program, a quasi-acronym for 
accountability, emphasis on basics and high standards, and maximum local control. The district 
decided to harmonize the Benchmark Goal Program with the state program for the 1997-98 
school year, but its major elements, described below, remained intact during the period of our 
study (the 1996-97 school year). 

In the original Benchmark Goal Program, the district set improvement goals for student 
achievement in nine areas: primary grade readiness, absenteeism, social studies, end-of-grade 
reading, writing, pre-algebra, dropouts, higher level course enrollment, and end-of-course subject 
matter mastery. Depending on grade-level composition, schools had between 14 and 44 sub- 
goals. Student achievement in cognitive areas was assessed by standardized multiple choice tests. 
In addition, goals were set for student absences, and for high schools, dropout rates and enroll- 
ment in higher-level courses. Initial performance baselines in the nine areas were established in 
1991-92, with annual improvement goals set for each subsequent school year. Every year an 
overall district improvement goal was set in each area (for example, the percentage of students 
scoring below the competent range on the writing test will decrease by five percent compared to 
the previous year). The district then set improvement goals for each school based on the per- 
formance level of the previous year’s students in each grade. Every school was given specific 
numerical improvement goals for each area. Maintenance goals also were established in areas in 
which the school was already performing at a high level, such as having 95 percent of its stu- 
dents prepared for pre-algebra. Maintenance goals were common for all students. There were 
separate improvement goals for white, African-American, and other students. 

Schools received points for achieving goals in each separate area. The central administration di- 
vided 100 points evenly among the school’s subgoals, so if a school had 25 subgoals, each sub- 
goal was worth four points. A school received 100 percent of the assigned bonus points for a 
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fully met subgoal, 75 percent of the bonus points for a partially met subgoal, and no bonus points 
for an unmet goal. Points earned on subgoals were summed to form a school’s total score. 
Schools that received 75 or more points were designated as “exemplary” and their certified staff 
received $1,000 and support staff $400. Schools that earned between 60 and 74 points were des- 
ignated as “outstanding.” Their certified staff received $750 and support staff $300. Schools that 
earned fewer than 60 points were given no special designation and staff did not receive a bonus. 

There were no formal sanctions for schools that failed to achieve the accountability goals in any 
one year. However, schools with chronically low achievement could be placed in the Priority 
Schools program. In 1995-96, 14 schools were so designated. With district assistance, these 
schools prepared special improvement plans and identified resources needed to improve student 
achievement. In addition, each school received assistance in staff training from a university team. 

The Benchmark Goals Program also contained a separate performance evaluation system for 
principals. The principal could earn up to 100 points annually, with 40 of those points based on 
school performance on the goals. Other evaluation factors included results of teacher, student 
and family surveys, facilities, and site-specific performance determined in consultation with the 
superintendent. Despite this separate evaluation system, principals were eligible to receive a bo- 
nus equivalent to the teacher amount if their schools were exemplary or outstanding. 

The Maryland School Performance Program 

The Maryland School Performance Program was adopted by the State Board of Education in 
1990 and is administered by the Maryland State Department of Education. This program holds 
schools accountable for academic improvement, monitors progress toward state standards, rec- 
ognizes school success with monetary awards, and assists or reconstitutes schools declining in 
performance. Rather than comparing schools and school systems, the performance of each school 
or district was measured in terms of its own growth toward the state’s long-term goals (Maryland 
Department of Education, 1997). To measure school performance, the State Department ofEdu- 
cation developed a School Performance Index. The School Performance Index is a weighted av- 
erage of a school’s relative distance from satisfactory standards for attendance and student per- 
formance on two test batteries: the Maryland School Performance Assessment Program and the 
Maryland Functional Test. The former is a statewide performance-based assessment used as the 
primary gauge of elementary and middle school performance. It was given in grades three, five, 
and eight. The Maryland School Performance Assessment Program was specifically designed to 
generate focused instructional change based on state learning outcomes (standards) as well as to 
monitor school performance. The performance target was for all schools to have 70 percent of 
students at or above satisfactory performance in all subjects covered by the Maryland School 
Performance Assessment Program by the year 2000. The Maryland Functional Tests are basic 
skills exams measuring competencies in reading, mathematics, writing, and citizenship and must 
be passed for high school graduation. The tests were given as early as grades seven and eight and 
are reported for grades nine and eleven. The long-term goal was that schools would achieve a 95 
percent pass rate in reading, an 80 percent pass rate in mathematics, and a 90 percent pass rate in 
writing by 2000. At the time of our study, a high school performance assessment was under con- 
struction and was scheduled to be tested in 1999. 
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The School Performance Index was calculated by averaging scores across the performance 
measures. For an elementary school, the Index was the average of the student attendance score 
and grade three and five achievement scores, the latter based on the percentage of students 
achieving the proficient level on the Maryland School Performance Assessment Program. A 
school’s ultimate goal was to reach 70 on the Index. The school recognition and rewards pro- 
gram was based on the improvement in performance over a two-year cycle. Change was meas- 
ured by subtracting the current year’s performance index from the average of the previous two 
years’ indices. (Averaging was intended to reduce the impact of fluctuations in the index on the 
change measure.) A school’s yearly goal was to make an improvement in the index that was 
judged by the Maryland Department of Education to be “sustained and significant.” “Significant” 
refers to statistically significant, and “sustained” to a pattern of improvement continuing over a 
number of years. 

Maryland elementary and middle schools that achieved a sustained and significant level of im- 
provement were eligible for a monetary bonus that could be used for school improvements, but 
not for staff pay bonuses. The amount of the award depended upon the size of the state appro- 
priation, the number of eligible schools, and the number of students in the reward schools. Be- 
tween 1996 and 1998, awards ranged from about $14,500 to $64,600 per school (Maryland De- 
partment of Education, 1999). To date, Maryland has allocated $8.25 million in monetary awards 
and has estimated that 10 percent of elementary and middle schools qualify for performance rec- 
ognition awards each year. Funds go to improving schools, not necessarily to schools where 
scores are high but are not improving. Non-monetary recognition certificates were awarded to 
schools that showed one year of significant improvement. The public reporting of scores also 
provided incentives in the form of public recognition for schools that improved, and sanctions, in 
the form of public exposure, for those schools that did not. The State Department of Education 
released a report card each December on state and district progress toward meeting the standards. 
The school districts were then responsible for reporting on individual school performance to their 
communities through the newspapers or other media (Maryland Department of Education, 1997). 

Schools that failed to improve faced a range of potential sanctions, from identification as being 
declared reconstitution-eligible to being identified as “at-risk of reconstitution,” to some form of 
reconstitution. Reconstitution could involve replacing the school’s administrative or teaching 
personnel and/or modifying the school’s organization or instructional program. Once the De- 
partment of Education identified a school as reconstitution-eligible, the school district had to re- 
spond by completing a local school reconstitution proposal, which outlines a framework to ad- 
dress those areas of decline. If failure persisted, the Department’s options included more drastic 
measures, such as contracting with a third party to run the school. At the time of our study (1997- 
98), this final action had not yet been pursued (Goertz, Massed, and Chun, 1997). To date, 97 
schools have been listed as reconstitution-eligible, with the majority in Baltimore City (Maryland 
Department of Education, 1999). For additional information on the Maryland program, see Kel- 
ley, Conley, and Kimball (1999). 
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MOTIVATIONAL MODEL 



Our initial research on the motivational effects of school-based performance award programs on 
teachers in Kentucky (described in Kelley and Protsik, 1997), using a grounded theory approach 
(Strauss and Corbin, 1994), suggested that Expectancy Theory (Vroom, 1964) and Goal Setting 
Theory (Locke and Latham, 1990) would be useful in understanding how teachers responded to 
the programs. Figure 1 shows the model of motivation, derived from Expectancy Theory and 
adapted to the SBPA context, that we then used to structure further research. The model depicts 
the requirements a successful award program must meet in order for teachers to be motivated to 
work toward improving student achievement. The central elements of the model are teachers’ 
perceptions of the degree to which the teacher’s own effort will influence student achievement, 
as measured in the program (called the expectancy perception), the teacher’s perception of the 
probability that meeting the student achievement goal will lead to the consequences (outcomes, 
such as receiving a bonus) associated with the program (called the instrumentality perception), 
and the positive and negative value (desirability) that they attribute to these outcomes or conse- 
quences. 

In this model, teachers’ motivation is a function of the expectancy, instrumentality, and desirabil- 
ity of outcomes. Starting with the expectancy link between effort and student achievement, the 
model shows that the overall effort expended by the teacher is derived from beliefs about how 
likely it is that effort (its intensity, focus, and persistence) will lead to the attainment of student 
achievement goals. The more a teacher perceives that working hard will pay off in meeting the 
goals, the greater the teacher’s motivation to expend the effort to do so. Moreover, borrowing 
from goal-setting theory (Locke and Latham, 1990), student achievement goals that are clear and 
specific will focus teachers’ efforts and foster stronger expectancy perceptions. 

Figure 1. Theoretical Framework: Teacher Motivation in 
School-Based Performance Award Programs 



Competencies 




Enablers 



In this model, teachers’ motivation is a function of the expectancy, instrumentality, and desirabil- 
ity of outcomes. Starting with the expectancy link (between effort and student achievement), the 
model shows that the overall effort expended by the teacher is derived from beliefs about how 
likely it is that effort (its intensity, focus, and persistence) will lead to the attainment of student 
achievement goals. The more a teacher perceives that working hard will pay off in meeting the 
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goals, the greater the teacher’s motivation to expend the effort to do so. Moreover, borrowing 
from goal-setting theory (Locke and Latham, 1990), student achievement goals that are clear and 
specific will focus teachers’ efforts and foster stronger expectancy perceptions. 

Teachers’ beliefs about the effort-student achievement linkage are influenced by their beliefs 
about their competencies and the presence of enabling conditions. Competencies refer to teacher 
knowledge and skills relevant to helping students meet achievement goals. Enabling conditions 
are organizational conditions that facilitate the meeting of student achievement goals. Examples 
include curricular resources, principal support, professional development opportunities, and 
feedback about progress toward meeting the goals. 

The instrumentality link (between meeting the goals and obtaining the outcomes) is also impor- 
tant in understanding the motivational impacts of school-based performance award programs. 
Instrumentality refers to teachers’ beliefs about the probability that meeting (or not meeting) stu- 
dent achievement goals will lead to the consequences or outcomes associated with the program, 
such as receiving a pay bonus or experiencing public criticism for failing to meet improvement 
goals. To the extent that teachers believe that consequences they value have a high probability of 
occurrence as a result of meeting the goals, they will be motivated to expend effort to meet the 
goals. Similarly, to the extent they believe that if the goals are not achieved, consequences they 
value negatively will occur, they will be motivated to try to achieve the goals. The point is that 
outcomes will motivate to the extent that teachers see them as highly likely to occur. 

The outcomes or consequences associated with the school-based performance award program 
must also have substantial negative or positive value to teachers in order to motivate them. Ex- 
amples of positively valued (desirable) consequences of SBPA programs include receiving a bo- 
nus, public recognition for meeting student achievement goals, and a sense of accomplishment 
from having students reach the achievement goals. Negatively valued (undesirable) conse- 
quences include not receiving a bonus, heightened job stress, experiencing public criticism for 
not meeting student achievement goals, and threats to job security. It should be noted that teach- 
ers are likely to differ as to whether a given consequence is seen as desirable or undesirable as 
well as the degree to which a consequence is perceived as desirable or undesirable. For example, 
though some teachers may negatively value stress, others may view stress positively, as a stimu- 
lant. The desirability or undesirability of each consequence is defined by the individual teacher. 
The importance of the desirability or value of outcomes or consequences is also recognized in 
the Goal Setting theory; goals associated with highly desirable outcomes are likely to attract a 
higher degree of acceptance and commitment. 

The theoretical framework illustrated in Figure 1 implies that attempts to motivate teachers to 
improve student achievement by an awards program is a complex undertaking. Program design- 
ers need to ensure that the outcomes associated with reaching accountability goals are highly 
valued by teachers and are perceived as highly likely to occur if the goals are reached. Program 
designers must also try to make sure that some likely consequences of trying to reach the goals 
(such as higher stress) are not so negatively valued as to counteract the likely positive conse- 
quences of reaching them. Teachers must also believe that if they expend effort to achieve the 
goals, their effort will bring results. This may require action aimed at improving teachers’ 
capacities or competencies and providing enabling conditions to support their efforts. School- 
based performance award programs are likely to motivate teachers when all of these 
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performance award programs are likely to motivate teachers when all of these requirements are 
addressed. Teachers in states, districts, or schools where these requirerrients are better addressed 
should be more motivated by these programs. 

We used this motivational model as the framework for our research to identify important vari- 
ables to study and relationships to test. Our study sought to address two cormected issues: what 
are the motivational responses of teachers to school-based performance award programs, and are 
teachers’ motivational responses to these programs predictive of schools’ success in meeting 
subsequent student achievement goals? The next section describes the way we explored these 
issues. 



STUDY METHODOLOGY 



Research Design 

This study examined school-based performance award programs in three jurisdictions: the state 
of Kentucky, the Charlotte-Mecklenburg district in North Carolina, and the state of Maryland. 

All three jurisdictions had comprehensive standards-based reform strategies in place. Kentucky 
and Charlotte-Mecklenburg were selected for the study because they paid salary bonuses to all of 
the teachers in a school for improvements in student outcomes toward a specified goal. We 
wanted research sites in which an awards program had already been implemented so that teach- 
ers had accumulated experiences with it. Maryland was chosen as a contrasting site because it 
provided awards to schools as school improvement funds not directly to teachers in the form of 
salary bonuses. 

Figure 2 provides an overview of our general research design, showing the basic steps we fol- 
lowed in Charlotte-Mecklenburg and Kentucky. 

Figure 2. Overview of the Research Process in Charlotte-Mecklenburg and Kentucky 




Motivational Responses of Teachers to Teacher Motivational 

School-Based Performance Award Programs Responses Used to 

Predict School Success 
in Meeting Goals 



We conducted our research in each jurisdiction between the begirming and end of a single pro- 
gram cycle. This allowed us to frame our interviews and survey questions, particularly regarding 
schools’ student achievement goals, in terms of that performance cycle. After determining our 
research sites and samples, we began by conducting interviews with teachers and principals from 
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a wide range of schools, varying in student socioeconomic status, grade level, and previous suc- 
cess in meeting student achievement goals. We used a semi-structured interview protocol based 
on the teacher motivation model. We specifically asked questions about expectancy, instrumen- 
tality, desirability of consequences or outcomes, competencies, and enabling conditions. We re- 
corded the interviews, transcribed them, and content analyzed the responses. Our content analy- 
sis of the responses was our first assessment of teachers’ motivational responses to the school- 
based performance award programs. 

Based upon what we learned from the interviews, we constructed surveys for administration to 
large, representative samples of teachers and principals in Charlotte-Mecklenburg and Kentucky, 
and of principals in Maryland. The primary content of the teacher survey focused on the five 
components of the teacher motivation model. From this survey we learned about the relative de- 
sirability of 17 outcomes (including the bonus) to teachers, the perceived probability of receiving 
these outcomes if a school met its goals, the perceived probability that teacher effort would result 
in a school meeting its goals, teacher perception of their competencies, and teacher perceptions 
of the presence of various enabling conditions in their schools. These responses allowed us to 
examine how well the awards program was functioning in generating positive motivational re- 
sponses from teachers. These interpretations of the survey data formed our second assessment of 
teachers’ motivational responses to school-based performance award programs. 

We also wanted to know whether variations in motivational responses of teachers were related to 
school success in meeting its student achievement goals at the end of the performance cycle. In 
this analysis we aggregated teachers’ motivational responses to the school level and used these 
aggregates to predict how well schools met their goals. In essence, we viewed the average level 
of motivation of teachers within each school as a predictor of whether or not their school met its 
goals. We expected that higher average levels of teacher motivation would lead to a greater like- 
lihood of goal attainment. We also recognized that the strength of the relationship between 
teacher motivation and school achievement we were likely to find might be weakened by many 
factors, including unreliability in the motivation and achievement measures and the impact of 
many factors other than teacher motivation on student achievement. However, these results 
would be our best answer to the question of whether teacher motivation leads to school achieve- 
ment success. 

In summary, in each jurisdiction, data collection involved on-site interviews with principals and 
teachers in schools with varying degrees of success in meeting school performance goals, fol- 
lowed by mail surveys of teachers (in Charlotte-Mecklenburg and Kentucky) and principals (in 
all three jurisdictions) in a larger sample of schools. Table A 1 in the Technical Appendix sum- 
marizes the nature and timing of the data collection effort. The Technical Appendix also includes 
a discussion of sampling designs, survey response rates, and other data collection issues. 
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FINDINGS: MOTIVATIONAL RESPONSES 
OF TEACHERS 



Perceptions About School Performance Goals 

According to Goal Setting Theory (Locke and Latham, 1990), in order for teachers to be moti- 
vated to expend effort to meet student achievement goals, they need to understand the goals and 
be committed to them. We expected school-based performance award programs to have positive 
impacts on both the degree to which teachers understand the goals and on goal commitment. 

Our interviews with teachers and principals suggested that the Charlotte-Mecklenburg and Ken- 
tucky programs were reasonably successful in getting teachers to understand and be committed 
to achieving the goals. All of the teachers we interviewed in both jurisdictions were aware of and 
had a significant degree of understanding of program goals. This is in contrast to our findings in 
Maryland, where a number of teachers we interviewed were unaware of the incentive program or 
poorly informed about the details. In both Kentucky and Charlotte-Mecklenburg, many teachers 
indicated that the presence of the award program provided them with clear direction and focus. 
Accountability and focus on the goals was named by many principals and teachers alike as the 
major positive effect of the programs in both Charlotte-Mecklenburg and Kentucky. While 
teacher commitment to these goals varied across schools, grade levels, and subjects, all of the 
teachers we talked to indicated that they had made changes in teaching practice to align curricu- 
lum and instruction with program goals. This finding is consistent with other research in Ken- 
tucky. For example, Stecher and Barron (1999) found that teachers in assessed subjects and 
grade levels had greater alignment of curriculum to the standards compared to teachers in sub- 
jects or grade levels where there was no assessment. In the most successful schools, changes to 
teaching practice appear to have occurred schoolwide. Despite these substantial effects, some 
teachers expressed a degree of conflict between the program’s and other goals. Competing goals 
mentioned were the need to emphasize basic skills, the importance of instilling a “love of learn- 
ing,” or pursuing a different program emphasis, such as at a fine arts magnet school (Heneman, 
1998; Kelley, 1998b). 

Our survey results were for the most part consistent with these interview findings. Teachers in 
both jurisdictions were asked about their commitment to achieving the goals, about how well 
they understood the goals, and whether these goals conflicted with other important educational 
goals. Teachers in Kentucky were also asked to agree or disagree with two statements about the 
focus or guidance provided by the goals. Table 1 shows the means and standard deviations on the 
scales we developed from these items. (Technical Appendix Table A2 provides for information 
on the items included in the scales and their reliabilities.) 



Table 1. Means and Standard Deviations of Responses on Goal-Related Scales 





Charlotte- 

Mecklenburg 

Mean 


Charlotte- 
Mecklenburg 
Std. Deviation 


Kentucky Mean 


Kentucky Std. 
Deviation 


Goal Commitment 


3.9 


0.8 


3.9 


0.7 


Goal Clarity 


3.4 


1.0 


3.6 


0.8 


Goal Focus 


- 


- 


3.3 


0.9 


Goal Conflict 


2.8 


1.0 


3.2 


1.1 



Scales are averages of responses to items with rating scales of ! -strongly disagree to 5=strongly agree. AH of the differences 
between Charlotte-Mecklenburg and Kentucky means are statistically significant at the .05 level or beyond. 
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On the average, teachers reported a high level of commitment to the goals. The averages for both 
Charlotte-Mecklenburg and Kentucky teachers on our commitment measure was 3.9 on a 1 to 5 
scale. Teachers also reported a reasonably high level of understanding of the goals, though the 
Kentucky average of 3.6 (on a 1 to 5 scale) was slightly higher than the average of 3.4 for Char- 
lotte-Mecklenburg teachers. There was less agreement by Kentucky teachers that the goals pro- 
vided a focus for teaching efforts. On the average, the level of agreement with the statements 
making up this two item scale was, at only .3 above the neutral point, somewhat lower than was 
expected based on what we heard in interviews with Kentucky teachers. On average, teachers in 
both jurisdictions reported some degree of conflict between the program goals and other impor- 
tant educational goals, though the fact that the averages were close to the neutral point suggests 
that this was not, overall, a major problem. (All of the differences in means across jurisdictions 
are statistically significant at the .05 level or beyond. However, it should be borne in mind that 
small differences in means will be statistically significant given the relatively large samples we 
used. This holds true for most of the other mean differences shown in subsequent tables.) 



Teachers in both jurisdictions also were asked about the level of performance they were aiming 
to achieve during the accountability cycle. There were asked specifically whether they were try- 



ing to achieve school performance levels that would lead to a reward, to avoid declines in student 
performance, or were they not particularly trying to meet the goals. Table 2 shows the percent- 
ages of respondents who indicated they were trying to attain the different levels of achievement 
established by each program. 

Table 2. Percent of Respondents Indicating They Tried to Achieve Various Performance or Goal Levels in 

Charlotte-Mecklenburg and Kentucky 


Charlotte-Mecklenburg 

During this year, did you: 


All Respondents 


Respondents in 
Schools Rewarded in 
Prior Year 


Respondents in 
Schools Not Rewarded 
in Prior Year 


not try to achieve goals 


5.6 


3.5 


7.5 


try to avoid decline in student 
performance measures 


7.2 


7.5 


7.0 


try to achieve Outstanding status 
(partial reward ) 


13.4 


9.3 


17.1 


try to achieve Exemplary status 
(full reward) 


73.8 


79.6 


68.4 


Kentucky 

During this year, did you: 


All Respondents 


Respondents in 
Cycle 2 Reward 
Schools 


Respondents in Cycle 
2 Nonreward Schools 


not try to achieve goals 


1.5 


2.2 


1.2 


try to avoid decline in KIRIS scores 


10.0 


7.3 


11.6 


try to achieve Improving Status 


10.8 


6.3 


13.6 


try to achieve Successful Status 


24.4 


12.0 


32.2 


try to achieve Reward Status 


53.4 


72.2 


41.4 



In both jurisdictions, the vast majority of teachers indicated they were trying to reach their 
school’s student achievement goals. Over 87 percent were trying for Outstanding or Exemplary 
status in Charlotte-Mecklenburg, and over 77 percent for Successful or Reward status in Ken- 
tucky. Few teachers indicated that they were not trying or were trying only to avoid performance 
declines. Moreover, as one might expect, meeting school goals in the last accountability cycle 
had an effect on the goal level teachers reported that they were aimed to achieve in the current 
period, particularly for schools in the highest prior achievement (and bonus- winning) level. 
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Teachers in Charlotte-Mecklenburg were more likely to report trying for Exemplary status if 
their school had earned a bonus the prior year (79.6 percent versus 68.4 percent in schools not 
earning a bonus last year). The effect of prior wiiming was even greater among Kentucky teach- 
ers, with Reward status being sought by more teachers in schools that had won (72.2 percent) 
than had not won (41 .4 percent) a bonus for the previous cycle. Another point of interest is that 
relatively fewer Kentucky teachers indicated that they were not trying to achieve the goals com- 
pared to Charlotte-Mecklenburg teachers, perhaps because the Kentucky program had stronger 
sanctions. 

The use of bonuses may help to increase teachers’ understanding of and commitment to goals by 
calling attention to the goals and signaling their importance. To investigate this possibility, we 
multiplied the desirability rating of the bonus by its perceived probability of occurrence if goals 
are met (instrumentality) to create the expected value of the bonus. Goal commitment and goal 
clarity were then each regressed on this expected value and on several control variables for Char- 
lotte-Mecklenburg and Kentucky survey respondents. We obtained significant positive coeffi- 
cients for the effect of the expected value of the bonus on goal commitment in both jurisdictions 
and for goal clarity in the Charlotte-Mecklenburg sample. (The goal clarity coefficient in the 
Kentucky analysis just missed significance at the .10 level.) For Kentucky, we also assessed the 
impact of the expected value of the bonus on goal focus, and found a positive and significant 
coefficient. Appendix Table A3 shows the coefficients. These results confirm the positive role 
that bonuses can play in getting teachers to understand and accept the goals. A stronger test, 
however, would be to compare goal clarity and commitment between programs with and without 
a bonus component. 

Because policy implementation research has repeatedly underscored how difficult it is for policy 
change to penetrate teaching practice (Fullan, 1991; Odden, 1991), these results are encouraging. 
In both jurisdictions with award programs that paid bonuses to teachers for goal attainment, there 
was definite penetration of the bonus-linked goals to the teachers in the form of goal commit- 
ment and understanding of the goals. 

Though each program seems to have succeeded in communicating expectations and securing 
commitment toward improvements in student achievement, one threat to goal commitment over 
time in both programs is the increasing difficulty of meeting ever-rising goals for successful 
schools. Both the Charlotte-Mecklenburg and Kentucky programs measure school performance 
in terms of improvement in student achievement. For schools that do improve, the absolute level 
of student achievement that must be produced to meet future goals will be higher in each suc- 
ceeding accountability cycle. For schools that do not improve in a cycle, the goal for the next 
cycle does not increase much, if at all, in terms of absolute student performance. We wondered if 
teachers in successful schools recognized that the goals were likely to get harder each year. In 
each jurisdiction, teachers were asked whether they thought the goals were harder this cycle than 
last, and would be harder yet next cycle. Table 3 shows the percentages of teachers who indi- 
cated that future goals would be more difficult, by the award status of their school in the last ac- 
countability cycle. In both jurisdictions, teachers in award schools were more likely to indicate 
that goals would get harder than teachers in non-award schools. 
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Table 3. Percentages of Teachers Indicating Goals Were Becoming Harder to Achieve, 



by School’s Award Status in Prior Cycle 


Percent of Charlotte-Mecklenburg respondents indicating that: 


Award Prior Cycle 


No Award 
Prior Cycle 


This year’s goals are much or somewhat harder 
Next year’s goals will be much or somewhat harder 




67.4 

59.5 


43.5 

44.1 


Percent of Kentucky respondents strongly agreeing or agreeing 
that: 


This cycle’s goals will be harder to achieve 




73.5 


63.5 


Next cycle’s goals will be even harder to reach 




70.9 


65.7 



Desirability of Outcomes 

Teachers’ interview responses provided several insights about what teachers perceived as out- 
comes associated with school-based performance award programs and their desirability (see 
Heneman, 1998; Kelley, 1998a; Kelley and Protsik, 1997). Teachers identified a number of out- 
comes, both extrinsic and intrinsic, that they experienced as a result of working under an award 
program. While many teachers indicated that bonuses for schoolwide improvement in student 
performance were appropriate, just as bonuses were appropriate for other types of professionals, 
many of the teachers said that they were not directly motivated by the promise of a bonus. In 
Charlotte-Mecklenburg, a number of teachers complained that the size of the bonus was too 
small ($400 to $600 after taxes) given the additional work required to achieve the bonus. This 
was particularly problematic given the overall low salaries, which required many beginning 
teachers to moonlight in order to support themselves. 

The interview data suggested that there were a variety of other important outcomes that teachers 
associated with participation in the school-based performance award program. Teachers identi- 
fied things like public recognition and professional pride as outcomes that motivated them to 
achieve program goals. They also mentioned that the changes they had made to curriculum and 
instruction were important improvements in their teaching practice. As a result of working to 
achieve program goals, teachers said they had improved their own teaching skills in ways that 
would make them better teachers, whether or not the award program continued. Teachers indi- 
cated that the programs provided new and valued opportunities for professional development, for 
working with other teachers, and for seeing measured improvement in student performance over 
time. 

Kentucky teachers were very aware of the sanctions component of the program and saw it as a 
real potential outcome for any school that did not improve performance. This was in sharp con- 
trast to the responses of Maryland teachers who accurately assessed that most schools were not 
in danger of sanctions. (The Maryland program identified schools for sanctions based on a com- 
bination of performance level and changes in performance, so only the lowest-performing 
schools in the state, mostly in Baltimore city, were likely to be sanctioned.) As a result, the sanc- 
tions component was not generally viewed as an important or motivating outcome for most 
teachers. Kentucky teachers indicated that they were motivated by the sanctions component out 
of fear of losing professional autonomy (due to the assignment of a Distinguished Educator to 
schools with declining performance), losing professional pride, and for some, a potential loss of 
job security. 
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Based on the teacher interview findings, we identified 17 potential outcomes directly relevant to 
experiences under the award programs, which we then included in the questionnaires sent to 
Charlotte-Mecklenburg and Kentucky teachers. Teachers rated the desirability of each outcome 
on a 1 to 5 scale. The average desirability ratings of the outcomes are shown in Table 4 (see also 
Heneman and Milanowski, 1999). 

Receiving a bonus was rated quite high in desirability, as were outcomes pertaining to recogni- 
tion, satisfaction in meeting goals, seeing students learn and improve performance, and working 
with other teachers. These results were consistent with our interview findings, confirming that 
under a school-based performance award program teachers find numerous outcomes desirable, 
and teachers find both extrinsic (such as receiving the bonus) and intrinsic (such as opportunities 
for professional development) outcomes desirable. Analysis of the ratings also showed that 
teachers who rated the bonus as more desirable did not rate as less desirable outcomes like per- 
sonal satisfaction from achieving the goals, personal satisfaction from improved student 
achievement, and seeing students learn new skills. 

Table 4 shows that teachers rated other outcomes quite low in desirability. These undesirable 
outcomes included public criticism for not meeting goals, loss of professional pride for not meet- 
ing goals, risk to job security, intervention, putting in more hours, less freedom to teach things 
unrelated to goals, and more pressure and job stress. In Kentucky, the intervention and risk to job 
security outcomes were rated about as undesirable as the positive outcomes associated with the 
program were rated desirable. This suggests the potential motivating strength of these sanctions. 
Avoiding these negative outcomes was not rated as desirable as the positive outcomes by Char- 
lotte-Mecklenburg teachers, which is likely a reflection of the more powerful negative sanctions 
program in Kentucky. These results are also consistent with the interview findings. 

To determine if there were any meaningful patterns or clusters of intercorrelations among the 
desirability ratings, we factor-analyzed (principal components, varimax rotation) the desirability 
ratings. Our interpretation of the factor loadings yielded the four groupings shown in Table 5. 

We labeled these “Goal Attainment Rewards,” “Learning,” “Sanctions,” and “Stress.” (One item 
relating to participation in meaningful education reform was discarded because of high cross- 
loadings across the factors.) These factors provide an intuitively meaningful way to summarize 
the desirability of the outcomes and largely reflect the distinctions between positive rewards and 
negative sanctions contingent on school performance. 

Outcome desirability ratings were not strongly related to teachers’ age, years of experience, sal- 
ary level, school level, percent of students receiving free and reduced price lunch, and whether a 
school had met its goals in the previous award cycle. There were very few statistically significant 
correlations between desirability ratings and these variables were obtained, and those that were 
significant were very small (i.e., r < .15). Thus, these desirability ratings appear to be generaliz- 

able across teachers within each jurisdiction. 
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Table 4. Desirability*^* of Outcomes Associated with School-Based Performance Award Programs 



Outcome 


Charlotte- 

Mecklenburg 

Mean 


Charlotte- 
Mecklenburg 
Std. Deviation 


Kentucky 

Mean 


Kentucky 

Std. 

Deviation 


Receiving a bonus'°^ 


4.5 


0.7 


4.1 


1.1 


Receiving funds for school improvements 


4.4 


0.7 


4.1 


0.9 


Public recognition for meeting goals 


4.1 


0.9 


4.0 


0.9 


Personal satisfaction of meeting goals 


4.3 


0.8 


4.2 


0.8 


Personal satisfaction from improved student 
Performance 


4.6 


0.6 


4.4 


0.6 


Participation in education reform program 


3.6 


1.0 


3.2 


1.1 


Having clear schoolwide goals 


4.4 


0.7 


4.3 


0.7 


Working cooperatively with other teachers on 
Curriculum and instruction 


4.3 


0.7 


4.0 


0.8 


Having your students learn new skills 


4.5 


0.6 


4.4 


0.7 


Having additional opportunities for professional 
Development 


4.3 


0.7 


3.9 


0.8 


Public criticism or embarrassment due to not achiev- 
ing goals^^^ 


4.4 


1.2 


1.6 


0.8 


Loss of professional pride due to not achieving 
goals^^^ 


3.6 


1.2 


1.8 


0.8 


Risk to job security^^^ 


3.9 


1.1 


1.6 


0.8 


Intervention^^^^^^ 


3.7 


1.3 


1.8 


0.8 


Putting in more hours 


2.0 


0.9 


1.9 


0.9 


Less freedom to teach things unrelated to goals 


2.0 


0.9 


1.9 


0.9 


More pressure and job stress 


1.6 


0.8 


1.5 


0.8 



(a) Scale: /= highly undesirable. ..5 = highly desirable. 

(b) $1,000 in Charlotte^ Mecklenburg. $2,000 in Kentucky 

(c) Charlotte-Mecklenburg respondents were asked to rate the desirability of avoiding these outcomes; Kentucky respondents 
were asked to rate the desirability or undesirability of the outcomes themselves. 

(d) Charlotte-Mecklenburg respondents were asked about having their school designated a Priority School; Kentucky respon- 
dents were asked about having a Distinguished Educator assigned to their school. 

All differences between comparable Charlotte-Mecklenburg and Kentucky averages are statistically significant at the .05 level or 
beyond. 



Table 5, Grouping of Outcome Items Based on Factor Analysis of Desirability Ratings 



Goal Attainment Rewards 

public recognition for meeting goals 
personal satisfaction from meeting goals 
receiving a bonus for meeting goals 
receiving school improvement funds for meeting goals 

Learning 

working cooperatively with other teachers 
working toward clear schoolwide goals 
having students learn new skills 
additional opportunities for professional development 
personal satisfaction from improved student perform- 
ance 



Sanctions 

public criticism and embarrassment for not meet- 
ing goals 

risk to job security 

loss of professional pride from not meeting goals 
intervention 

Stress 

more pressure and job stress 
putting in more hours 

less freedom to teach things unrelated to the 
goals 



Combining the interview and survey data, we clearly and convincingly found that a school-based 
performance award program is not just a bonus program; rather, it is one element in an interre- 
lated system of rewards, opportunities, and demands that influences teachers’ jobs and lives in a 
multitude of ways, leading them to experience (and form values about) a variety of both extrinsic 
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and intrinsic outcomes. However, it is the bonus part of the program that is distinctive (and 
costly), so teachers’ reaction to it are especially important. We found that most teachers viewed 
receiving a bonus per se as appropriate, desirable, and not in conflict with other outcomes tradi- 
tional seen as motivating to teachers. We were surprised to find less desirability and motivation 
being reported for the bonus program as a whole. This inconsistency seemed to be a reflection of 
unfavorable attitudes about components of the school-based performance award programs, and 
how they are administered, rather than about the bonus per se. This is discussed further below. 

We were also somewhat surprised by the number of relatively undesirable outcomes that teach- 
ers mentioned, and the intensity of some teachers’ feelings about them. Increased work hours, 
pressure, and stress stood out in this regard. In addition, for many Kentucky teachers, the job se- 
curity and intervention threats were perceived as intense. Some teachers, however, viewed the 
longer hours and “being under the gun” as necessary to improve student achievement and moti- 
vate the shifts in practice required. Also, some teachers who had experienced state intervention 
viewed it as professional growth experience that brought new resources and a sense of focus to 
the school. 

Instrumentality Perceptions 

In the interviews, we asked teachers about the likelihood of various outcomes occurring if their 
school met, or did not meet, its student achievement goals. We concentrated particularly on the 
bonus, risk to job security, and intervention. We also asked about the likelihood of outcomes oc- 
curring simply as a consequence of trying to reach the student achievement goals. We found 
some degree of skepticism that the bonus would be paid if a school met its goals. This skepticism 
was most pronounced in Kentucky and may have been a reflection of teachers’ experience years 
earlier when the state promised, but then never paid, a $300 bonus payment to all teachers. The 
Kentucky teachers also were skeptical about the long-term survival of the award program. The 
Charlotte-Mecklenburg teachers, although less skeptical, questioned whether there would be 
long-term funding for the bonus payments, as well as the longevity of the program itself (Some 
level of skepticism was probably justified in both jurisdictions since both programs were 
changed significantly after our research was completed.) Compared to Charlotte-Mecklenburg 
teachers, the Kentucky teachers saw losing their jobs or having an intervention occur as much 
more likely, a reflection of the strong negative sanctions not only possible, but sometimes used, 
in Kentucky. Kentucky teachers also appeared to perceive that program participation was more 
likely to lead to increased pressure and stress. They cited loss of professional autonomy as a 
more likely outcome (Kelley, 1998a; Kelley and Protsik, 1997) than teachers in Charlotte- 
Mecklenburg or Maryland did. This may have been due to the more difficult goals with which 
many Kentucky teachers were confronted, as well as the more stringent sanctions that could be 
imposed for poor performance on the accountability index. 

Our survey asked teachers about the likelihood of each of the 17 outcomes. For some outcomes 
(e.g., receiving a bonus), teachers were asked to indicate the probability of the outcome occur- 
ring if their school met its goals. For others (e.g., loss of job security), they were asked to indi- 
cate the likelihood of that outcome occurring if the school did not meet its goals. For outcomes 
such as increased work hours and more job stress, they were asked the probability of experienc- 
ing the outcome if trying to meet school goals. The probabilities were assessed on a 0 to 100 per- 
cent scale, in 10 percent increments. The means of the probabilities are shown in Table 6. 



CPRE Research Report Series, RR-44 



17 



School-Based Performance Award Programs, Teacher Motivation, and School Performance 



Table 6. Average Perceived Probability(a) of Outcomes Associated with School-Based Performance 

Award Programs in Charlotte-Mecklenburg and Kentucky 

Outcome Probability if School Probability if You Try to Probability if School 

Meets Goals Meet Goals Does Not Meet Goals 

Charlotte Kentucky Charlotte Kentucky Charlotte Kentucky 

Receiving a bonus^^^ 72.8 54.8 

Personal satisfaction from 79.4 73.9 

meeting goals 

Personal satisfaction from 78.9 73.6 

improved student 

Performance 

Public recognition for meet- 67.6 62.1 

ing goals 

Receiving funds for school 45.2 49.6 

improvements 



Working cooperatively with 
other teachers on curricu- 


72.9 


71.5 






lum and instruction* 

Feel you are working to- 
ward clear schoolwide 


73.6 


63.3 






goals 

Having additional opportu- 
nities for professional 
development 


63.5 


68.5 






Having your students learn 
new skills 


54.6 


59.9 






Feel you are participation in 
education reform program* 


58.0 


56.4 






More pressure and job 
stress 


73.6 


84.4 






Putting in more hours 


65.8 


81.9 






Less freedom to teach 
things unrelated to goals 


68.9 


74.9 






Public criticism or 






53.8 


68.6 


Embarrassment due to not 
achieving goals 
Intervention 






46.5 


75.9 


Loss of professional pride 
due to not achieving goals 






44.9 


61.0 


Risk to job security 






31.8 


50.2 



(a) Response scale ranged from 0 (Highly Unlikely) to 100 (Highly Likely) in 10 point steps. For standard deviations, see Tech- 
nical Appendix Table A4. 

(b) $1,000 in Charlotte-Mecklenburg, $2,000 in Kentucky. 

All differences between comparable Charlotte-Mecklenburg and Kentucky averages are statistically significant at the .05 level or 
beyond except where indicated by an asterisk (*). 



Except in the case of the bonus, there were only small differences between Charlotte- 
Mecklenburg and Kentucky teachers in the perceived probabilities of the other positive outcomes 
occurring if a school meets its goals. The perceived bonus probability was much higher (72.8 
percent) in Charlotte-Mecklenburg than in Kentucky (54.8 percent). The perceived probability of 
most of the positive outcomes associated with trying to meet school goals appeared quite similar 
for Charlotte-Mecklenburg and Kentucky teachers, but the latter saw a higher probability of 
more pressure and stress and working more hours. Perceived probabilities of outcomes associ- 
ated with failing to meet goals were substantially higher for all four outcomes for Kentucky 
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teachers, probably due to the formal sanctions provided by the program. In comparison to the 
Kentucky teachers, the Charlotte-Mecklenburg teachers perceived a much greater probability of 
receiving a bonus if school goals were met, and they perceived a much lower probability of ex- 
periencing sanctions if the goals were not met. 

In another set of analyses, we correlated probability perceptions with teachers’ age, years of ex- 
perience, salary, school level, school’s percent of students receiving free and reduced-price 
lunch, and whether a school met its goals in the previous award cycle. Most of these correlations 
were not statistically significant, and those that were significant were small (for example, r < 15). 
The one exception was a moderate and significant correlation between the probability of winning 
a bonus for meeting school goals and whether the school had met its goals (and teachers earned 
bonuses) in the previous award cycle (r = .23 for Charlotte-Mecklenburg teachers, r =.30 for 
Kentucky teachers, p < .05). 

The interview and survey findings converge and complement one another. The most striking 
finding in both Charlotte-Mecklenburg and Kentucky was the relatively low perceived probabil- 
ity of receiving a bonus even if school goals had been met. Moreover, there was only a small cor- 
relation between perceived probability of receiving the bonus for achieving the goals and having 
received a bonus in the past award cycle. These findings suggest a possible lack of credibility of 
the bonus portion of the programs. Even reaching goals and winning bonuses in the previous cy- 
cle was not a very strong influence on believing that history would indeed repeat itself A second 
striking finding was that Kentucky teachers, relative to Charlotte-Mecklenburg teachers, per- 
ceived much greater probabilities of occurrence of negative outcomes if they tried to meet the 
goals. Putting in more hours and more pressure and stress were seen as more likely than any of 
the positive outcomes. Kentucky teachers also saw a higher probability of sanctions for poor per- 
formance than Charlotte-Mecklenburg teachers, reflecting the stronger sanctions provided by the 
Kentucky program. Kentucky teachers appear to have gotten the message of the potential sanc- 
tions. Coupled with the low rated desirability of these outcomes, the perceived connection be- 
tween not improving and sanctions confirms their potential motivating power in Kentucky. 

Expectancy Perceptions 

Interview data from Kentucky and Charlotte-Mecklenburg suggest that characteristics of indi- 
vidual teachers and schools have an important effect on the level of teacher expectancy (the be- 
lief that teacher effort will lead to meeting program goals) and school success. When present, 
these characteristics, which we call “enabling conditions,” provide organizational contexts that 
can foster high levels of expectancy and high achievement gains. The interview data from Ken- 
tucky revealed that award schools differed from non-award schools in some important ways. 
Award schools shared the following perceived enabling conditions: high teacher and principal 
expectancy perceptions, principal and district support for the goals of the program, high quality 
professional development linked with the accountability program, meaningful collaboration 
among teachers, and connections to external professional communities. In non-award schools, 
teachers and principals tended to have lower expectancies about their efforts leading to success 
and multiple or conflicting goals that competed with the goals of the accountability program 
(Kelley, 1998a). 
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The teacher surveys administered in Charlotte-Mecklenburg and Kentucky were designed to fur- 
ther illuminate important organizational conditions that contribute to teacher expectancy. We 
measured both individual teacher expectancy and teacher group expectancy. Individual teacher 
expectancy is the extent to which a teacher believes that her own effort will result in achievement 
of school accountability goals. Teacher group expectancy is the extent to which an individual 
teacher believes that she and her colleagues working together can achieve the school account- 
ability goals. We measured these constructs by asking teachers to estimate the probability that if 
they (or they and their colleagues) worked hard, the school would achieve its goals to the extent 
needed to earn the performance award. The correlation between the two measures of expectancy 
was .88 in Charlotte-Mecklenburg and .69 in Kentucky. Table 7 shows the average expectancy 
scores for Charlotte-Mecklenburg and Kentucky teachers. 



Table 7. Average Individual and Group Expectancy: Kentucky and Charlotte-Mecklenburg Teachers 





Charlotte 


Charlotte 


Kentucky 


Kentucky 




Average*^ 


Std. 


Average * 


Std. 






Deviation 




Deviation 


Individual Expectancy-Achleve Award 


55.6 


29.5 


39.2 


26.8 


Group Expectancy-Achieve Award 


62.0 


29.1 


52.7 


27.3 



Response scale ranged from 0 (Highly Unlikely) to 100 (Highly Likely) in 10 point steps. 



All differences between comparable Charlotte-Mecklenburg and Kentucky averages are statistically significant at the .05 level 
or beyond 



Neither individual nor group expectancy levels were particularly high in either jurisdiction, but 
the averages were considerably lower in Kentucky. In Kentucky, teachers on the average did not 
see even a 50 percent probability that if as an individual they tried, their school would achieve a 
level of student performance necessary to achieve reward status. They were more positive, how- 
ever, about the likelihood that if they and their colleagues worked hard together, their school 
would achieve reward status. Because sanctions were an important issue in Kentucky, we also 
asked our Kentucky respondents about their expectancy of achieving a performance level suffi- 
cient to avoid sanctions. The average probability that individual effort would lead to sufficient 
performance to avoid sanctions was 44.5 percent, and the average group expectancy probability 
was 60.9 percent. These averages were not all that much higher than those for reward-winning 
performance. Because avoiding sanctions required only maintaining last cycle’s performance 
level, this suggests that many Kentucky teachers had serious doubts about the relationship be- 
tween their efforts and student achievement. 

Explaining Teacher Expectancy Perceptions 

We found that the level of expectancy varies greatly across teachers. According to our motiva- 
tional model, potential determinants of teacher expectancy include both teachers’ assessments of 
their competencies and their assessments of whether important enablers were present. We used 
the survey responses to explore the degree to which these factors influenced teacher expectancy 
in Kentucky and Charlotte-Mecklenburg (see also Kelley, 1998a). Four kinds of factors were 
used to try to explain expectancy perceptions: school characteristics, organizational context, 
teacher attitudes, and teacher characteristics representing competencies. School variables were 
included to control for the effect of differences in school level and prior experience with the re- 
ward program (achieving award status or not), school size, and the percent of students receiving 
free or reduced-price lunch. 
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Organizational context variables included feedback, alignment, professional community, teacher 
participation in decision-making, principal support, and goal conflict. All context variables rep- 
resented equivalent constructs across jurisdictions, except that we added more professional 
community items to the Kentucky survey, and these were included in an alternative professional 
community variable for the Kentucky analysis. Teacher attitude variables included teacher per- 
ceptions of program fairness and goal clarity. Teacher competency variables included years of 
teaching experience and highest level of education completed. A description of the items in- 
cluded in the various scales used in the analysis is found in Technical Appendix Table A5. 

Tables 8 and 9 present the results of the regression analyses for individual and group expectancy 
in Charlotte-Mecklenburg and Kentucky. (Means, standard deviations, and intercorrelations of 
the variables can be found in Technical Appendix Table A6.) Variables found to have a signifi- 
cant effect on expectancy in both jurisdictions include feedback, goal conflict, school manage- 
ment, leadership and culture, and the perceived fairness of the program. Professional community 
was measured in two ways in the Kentucky analysis. In Table 9, results of two alternative analy- 
ses are reported, each using a different version of the professional community variable. The vari- 
able labeled “Community (a)” was a single-item that was the same as used in the Charlotte- 
Mecklenburg analysis, while that labeled “Community (b)” was a multiple item measure drawn 
from the work of Newman and Associates (1996). 

Table 8. Regression Estimates (Standardized Coefficients) of the Effects of Factors Affecting Individual 
and Teacher Group Expectancy Perceptions in Charlotte-Mecklenburg's School-Based Performance 



Award Program (N=900) 


Variable 


Individual Teacher Expectancy 


Teacher Group Expectancy 


School Characteristics 






High School 


-.235*** 


-.187** 


Middle School 


-.083** 


-.020 


School Size 


.060 


.040 


Free Lunch 


-.023 


-.044 


Award History 


.031 


.022 


Organizational Context 






Feedback 


.175*** 


.171*** 


Alignment 


-.016 


.056 


Community 


.021 


-.064 


Participation 


-.002 


-.006 


Principal Support 


.110** 


.032 


Goal Conflict 


-.051 


-.062 


Teacher Attitudes 






Fairness 


.269*** 


.293*** 


Goal Clarity 


.048 


.059 


Teacher Characteristics 






Experience 


-.020 


-.042 


Education 


-.047 


-.028 




.237 


.215 


- .237 for individual expectancy and .215 for group expectancy. 



*** Statistically significant at the .01 level 



** Statistically significant at the . 05 level 
* Statistically significant at the .10 level 




r\ 

C 



1 
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Among the organizational context variables, feedback had the strongest effect on individual 
teacher and teacher group expectancy. Feedback involves the receipt of the results of past as- 
sessments, and capacity to interpret and use this information to refine curriculum and instruc- 
tional programs. Goal conflict had a consistently significant dampening effect on individual 
teacher and teacher group expectancies. Goal conflict is the presence of organizational goals that 
compete with the goals of the accountability program. Competing goals can have a variety of 
sources. During our interviews, we found that schools with academic programs ranging from 
very weak to very strong had other goals besides doing well in the accountability program. For 
example, teachers in magnet schools indicated that the presence of a strong magnet emphasis 
limited their ability to focus on the goals of the accountability program. 



School management, leadership, and culture variables also had important effects on individual 
teacher, and teacher group expectancies. Principal support had a positive effect on expectancies 
in Charlotte-Mecklenburg, and professional community had a positive effect in Kentucky. Pro- 
fessional community may have a stronger impact in Kentucky because the accountability pro- 
gram had a slightly different focus. Some of the goals in Charlotte-Mecklenburg involved mak- 
ing structural changes, such as increasing course enrollments in higher level courses. High levels 
of professional community may not be required to successfully accomplish this type of goal. 

Table 9. Regression Estimates (Standardized Coefficients) of the Effects of Factors Affecting Individual 

and Teacher Group Expectancy Perceptions in Kentucky’s School-Based Performance Award Program 

(N=1,384 or 1,393) 

Variable Individual Teacher Expectancy Teacher Group Expectancy 

School Characteristics 



High School 


-,074* 


-.072* 


-.033 


-.030 


Middle School 


-.036 


-.029 


-.061* 


-.059* 


School Size 


-.004 


-.013 


.096** 


.086** 


Free Lunch 


-.028 


-.040 


.015 


.007 


Award History 


,148*** 


.137*** 


.143*** 


.134*** 


Organizational Context 










Feedback 


.173*** 


CO 


.131*** 


.137*** 


Alignment 


-,016 


-.023 


.021 


.039 


Community (a) 


.112*** 




.037 




Community (b) 




.121***- 




-.002 


Participation 


.043 


.028 


-.015 


-.019 


Principal Support 


.010 


.034 


.063* 


.079** 


Goal Conflict 


-,102*** 


-,088*** 


-.180*** 


-.170*** 


Teacher Attitudes 










Fairness 


,205*** 


.233*** 


.293*** 


.307*** 


Goal Clarity 


-,038 


-,053 


-.022 


-.028 


Teacher Characteristics 










Experience 


,008 


.009 


.002 


-.005 


Education 


-.049 


-.042 


-.030 


-.025 




.221 


.229 


.291 


.296 



*** Statistically Significant at the .01 Level (a) Same question as in Charlotte-Mecklenburg. 

** Statistically Significant at the .05 Level (b) Augmented scale with additional questions. 

* Statistically Significant at the . 10 Level 



Among the teacher attitude variables, perceptions of program fairness was a strong predictor of 
expectancy. The fairness variable reflected the perception that the program was administered 
fairly and that each school had an equal chance of doing well in the program. Fairness responses 
may be important because they reflect teacher attitudes about the quality of the program and 
about how well the system as a whole can perform in meeting or exceeding accountability pro- 
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gram goals. The importance of the fairness variable suggests that communication and teacher 
involvement may be key factors to consider in developing award programs to ensure that teach- 
ers understand and are comfortable with the rules of the game. Measures of goal clarity, align- 
ment, and student socioeconomic status were not found to be significant predictors of teacher 
expectancies, nor were our measures of teacher competency, experience, and education level. 

Student socioeconomic status may be less important as a predictor of expectancy in these pro- 
grams because, by design, they define school goals in terms of improvement, so low socioeco- 
nomic status schools do not have to produce the same absolute student achievement as high so- 
cioeconomic status schools. 

Further study is needed to better understand the importance of some of these variables. For ex- 
ample, goal clarity and alignment may be very important influences on teacher expectancy, but 
they may not have shown up strongly in these analyses for two reasons. First, alignment and clar- 
ity are correlated with each other and with many of the other enabling conditions making it hard 
to get a precise estimate of their effects. Second, a relatively high level of clarity and alignment 
may be present in most schools due to the communicative effect of school-based performance 
awards themselves (Kelley, 1998b), the efforts both jurisdictions made to communicate about the 
program, and the pressure placed on schools to align their programs to these goals. If most 
schools already have fairly high levels of alignment and clarity, then their effects would not 
show up strongly here, unlike a study that compared expectancies across accountability programs 
with and without an award component. There also is a need for better measures of teacher 
knowledge and skills, which theory suggests would influence expectancy perceptions. Years of 
experience and education credits and degrees, as used here, are probably weak proxies for actual 
skills (Odden and Kelley, 1997). 

In sum, we found in both qualitative and quantitative data analyses that higher expectancy occurs 
when organizational incentives, resources, and values are aligned with the goals of the account- 
ability program. 

The Motivational Role of the Bonus 

Because of the central role of bonuses in school-based performance award programs and the lack 
of evidence available about their motivational impact on teachers, we explored teachers’ reac- 
tions to the bonuses in depth. From our interviews with teachers in the three jurisdictions, we 
learned that: 

• Teachers felt it was appropriate for teachers and classified staff to receive bonuses and 
that receiving a bonus was deserved. 

• Teachers had differing reactions to the size of the bonus. Some thought the bonus 
amounts were significant and meaningful, while others described them as inadequate af- 
ter deductions for taxes (especially in Charlotte-Mecklenburg) or in relation to the 
amount of additional work required to earn the bonus (especially in Kentucky). 
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• Teachers had varying views about how much the possibility of earning a bonus motivated 
them to improve student achievement or whether a larger bonus would motivate them 
more. 

• Teachers were skeptical that earned bonuses would actually be paid due to past experi- 
ences with undelivered bonus payments. They had doubts about whether the funding for 
the bonuses would be continued and doubts about policymakers’ continued commitment 
to the program. 

. Teachers differed as to whether they wanted the bonus part of the program to continue. 

. The meaning of the bonus varied. For some, it was a ‘thank you,’ or a formal recognition. 

For others, it was a reward that allowed for the purchase of desired goods, and for still 
others, it was simply irrelevant. 

• Teachers in Kentucky said that having to decide how to allocate the award money among 
teachers and staff was a divisive process that created tension within and between schools. 
(See also Abelmann and Kenyon, 1996.) 

. Awards paid as salary bonuses appeared to have more visibility than awards paid as 

school improvement funds. Of the three programs studied, teachers in Maryland were the 
least cognizant of the award program and least likely to report that they felt job pressure 
and stress as a result of the program. 

These interview findings, in combination with the survey findings that teachers valued the poten- 
tial bonuses while having doubts about receiving them even if school goals were met, suggested 
that teachers’ reactions to the bonus per se and to the award program as they experienced it were 
complex. Moreover, there were clearly strong differences in reactions among teachers to all fac- 
ets of the programs. To delve more deeply into the nature and causes of these reactions, and into 
possible teacher decisions to withdraw from being a part of a school-based performance award 
program, we conducted two additional sets of analyses. The first sought to predict teachers’ 
overall responses to the bonus program in terms of their level of motivation, whether they felt the 
bonus part of the accountability program should continue, and their probability of withdrawing 
from the program in various ways. The independent variables pertained primarily to various ele- 
ments of perceived fairness of the program. These analyses, described in more detail in Heneman 
and Milanowski (1999), were conducted using the survey responses of a subset of Kentucky 
teachers. We conducted a second set of analyses using Charlotte-Mecklenburg teachers to exam- 
ine factors predictive of their withdrawal from the program by leaving the district or transferring 
to another school (see also Heneman and Milanowski, 1998). The independent variables were 
salary, satisfaction with salary, whether the teacher was working in an award-winning school, the 
perceived fairness of the bonus, and if the teachers had an intention to leave his or her current 
job. 

Motivational Reactions of a Subsample of Kentucky Teachers 

We randomly selected half of the Kentucky teacher sample to receive additional specific ques- 
tions about their reactions to the bonus program. (These items are shown in Technical Appendix, 
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Table A7.) The first three sets of itemsDbonus motivation, desire to see the bonus continue, and 
withdrawal intentionsOconstituted the three dependent variables. Bonus motivation and with- 
drawal intentions were measured on five-point rating scales (l=strongly disagree to 5=strongly 
agree) and summed to form a single scale. Favorability toward bonus continuation was measured 
on a three point scale (l=should not continue, 2= neutral, 3=should continue). The means were 
2.30 for bonus motivation, 3.09 for withdrawal intentions, and 2.03 for bonus continuation. The 
average level of the bonus motivation measure was more than .5 scale points below the 3.0 neu- 
tral point. Withdrawal intentions and the desire to see the program continue were slightly above 
the neutral point. (We also asked similar questions of the other half of the Kentucky teacher 
sample, and average response levels were similar.) 



The independent variables were derived from the literature on distributive justice, procedural jus- 
tice, and pay satisfaction (see Heneman and Judge, in press, for a review). Distributive justice 
refers to the fairness of the amount of reward that could be earned. Procedural justice refers to 
the fairness of the process used to administer the rewards. Pay satisfaction represents feelings 
about one’s actual pay and its fairness. We hypothesized that the distributive justice scales 
(named “fair bonus,” “expected bonus value,” and “teachers divide” in Table 10) and procedural 
justice scales (fair chance, fair process, principal communication) would be positively related to 
bonus motivation and desire to see the bonus continue, and would be negatively related to with- 
drawal intentions. We hypothesized that the other procedural justice scale, unfair wirmers, would 
be negatively related to bonus motivation and bonus continuation, and positively related to with- 
drawal intentions. Pay satisfaction was measured by the pay level scale of the Pay Satisfaction 
Questionnaire (Heneman and Schwab, 1985). We hypothesized that pay satisfaction would be 
negatively related to bonus motivation and the desire to see the bonus continue and negatively 



Table 10. Effect of Kentucky Teachers' Bonus Attitudes on Their Reported Motivation by Bonus, Support 
for Bonus Continuation, and Withdrawal Intentions (Standardized Regression Coefficients) 
(N = 747 to 754) 



Independent Variable 


Bonus 

Motivation 


Bonus Should 
Continue^ 


More Likely to 
Withdraw 


Years Experience 


-.173*** 


-.102** 


-.000 


Job Satisfaction 


-.050 


.043 


-.272*** 


Teach a KIRIS subject (1=yes) 


-.026 


.013 


.041 


Teach a KIRIS grade (1=yes) 


-.013 


.013 


.066 


High School 


.124*** 


.054 


.027 


Middle School 


.031 


.095* 


.051 


MA Degree or above 


.092** 


-.028 


.020 


Cycle 2 Bonus (1=yes) 


.093** 


.089 


.031 


Pay Satisfaction 


-.188*** 


-.165*** 


.001 


Fair Bonus 


.266*** 


.621*** 


.015 


Expected Bonus Value 


.050 


.079* 


.033 


Unfair Winners 


.020 


-.062 


.130*** 


Fair Chance 


.014 


.077 


-.091** 


Fair Process 


.092* 


.283*** 


-.237*** 


Teachers Divide 


.133*** 


.153*** 


-.006 


Principal Communication 


-.138*** 


-.067 


-.017 


R^ 


.220 


.275 


.458^ 



* significant at the JO level (a) Ordered logistic regression partially standardized coejficients 

** significant at the .05 level (b) pseudo-R^, based on the likelihood ratio 

*** significant at the .01 level 
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related to withdrawal intentions. To help avoid the possibility of spurious correlations between 
the independent and dependent variables, we incorporated several control variables into the 
analysis. These variables included years of experience, overall job satisfaction, whether the re- 
spondent taught a KIRIS subject, taught a KIRIS grade, was a high school or middle school 
teacher, whether the respondent held a master’s degree or above, and whether the school earned 
a bonus in last cycle. Means, standard deviations, and intercorrelation for the variables are shown 
in Technical Appendix Table A8. Regression results are presented in Table 10. 

These analyses produced several significant results. The teachers most motivated by the bonus 
were those who were dissatisfied with their current salary, felt the bonus amount was fair, felt the 
bonus process was fair, felt teachers should divide the bonus money, and perceived less commu- 
nication from the principal. Teachers in favor of seeing the bonus program continue also were 
more dissatisfied with their current salary, saw a high expected bonus value, felt that the bonus 
process was fair, felt that the bonus itself was fair, and felt that teachers should divide the bonus 
money. With the sole exception of the finding about principal communication, all of these results 
were as hypothesized. It should be noted, however, that no significant results were obtained for 
the unfair winners and fair chance variables. 

Teachers were most likely to intend to withdraw (leave their school or leave teaching) if they 
perceived there were unfair winners, that the process was not fair, and that their school did not 
have a fair chance of winning. Surprisingly, pay satisfaction, did not significantly relate to with- 
drawal intentions. 

To summarize, we found that the Kentucky teachers on average did not report being highly mo- 
tivated by the bonus program, but were more neutral in their intentions to withdraw from the 
program and in their desire to see it continue. These data, however, were collected at a time of 
contention and uproar about the KIRIS program (Blackford, 1997; Lawton, 1997; Strecklow, 
1997), and it is unknown whether more favorable attitudes would have been expressed had the 
data been collected earlier. There was considerable variability in teachers’ overall attitudes about 
Kentucky’s bonus program, and several hypothesized factors accounted for a significant portion 
of this variability. Teachers were more motivated by the program and wanted it to continue if 
they were dissatisfied with their current salary, felt that the use of bonuses and the process of 
awarding them were fair, and felt that teachers should decide how to divide up the bonus money. 
A different set of factors were related to teachers’ withdrawal intentions. Teachers were more 
likely to want to withdraw from their current school because of perceived unfairness in how the 
bonus program was run, an unfair chance for their school to meet its goal, and the existence of 
unfair winners. 

Teacher Withdrawal in Charlotte-Mecklenburg 

In response to dissatisfaction with their compensation experiences, teachers may opt to withdraw 
from their current job by leaving the district altogether (turnover) or by transferring to another 
school within the district (transfer). Compensation experiences encompass both current base pay 
and the bonus earned under the award program. To evaluate the impacts of these compensation 
experiences on these two forms of withdrawal, we gathered turnover and school transfer data 
from the Charlotte-Mecklenburg personnel data base in September, 1997, roughly five months 
after the teacher surveys had been completed. We then used the survey data to predict whether 
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teachers left the district (1 = yes), or transferred schools (1 = yes) using logistic regression analy- 
sis. We used five independent variables, four pertaining to compensation experiences and one 
pertaining to an overall intention to leave the current job and school for any reason, not just 
compensation experience. The first two variables were annual pay and an average of the three 
pay level items of the Pay Satisfaction Questionnaire (Heneman and Schwab, 1985). The second 
two variables were whether the teacher was working in a school that had earned an award in the 
previous year (1 = yes) and a rating (1 to 5) of the fairness of the bonus amount that could be re- 
ceived under the award program. The last variable was intent to leave, measured as the sum of 
the ratings (1 to 5) of the likelihood the teacher would leave because of the award program in 
order to pursue each of five alternatives (transfer to an award school, leave school for new teach- 
ing job, take an administrative job, leave for a job outside of teaching, retire). We controlled for 
several variables that might co-vary with withdrawal decisions: years in current job, age, gender, 
minority status, possession of an advanced degree, whether the teacher taught math or science, 
school level, and percentage of the school’s students receiving free or reduced price lunch. 

Means and standard deviations of the variables are shown in Technical Appendix Table A9. The 
total turnover rate was 19 percent with 1 1 percent due to school transfers and 8 percent due to 
leaving the district. The average pay level was $29,660 and the average satisfaction with pay 
level was quite low (mean = 1.61). On average, the teachers neither agreed nor disagreed that the 
amount of the bonus they could earn was fair (mean = 3.1 1), and 51 percent of the teachers 
worked in a school that had won an award the previous year. The average intent to leave one’s 
current job within the next two years because of the award program (mean = 2.61) was slightly 
below the scale midpoint. 

Logistic regression results are shown in Table 1 1 for each of the three dependent variablesDtotal 
turnover (leaving the district or transferring between schools), leaving the district, and transfer- 
ring between schools. 

Pay level was negatively related to all three types of turnover, though the coefficients in the 
analyses of leaving the district and transferring between schools just missed being significant. 
There was an unexpected positive, significant coefficient for pay level satisfaction on total turn- 
over and leaving the district. Non-bonus schools had significantly more total turnover due to a 
greater number of teacher transfers out of those schools. Intent to leave within the next two years 
because of the award program had the strongest impact of all the variables on total turnover, 
leaving the district, and transferring schools. 

It is important to note that the obtained relationships between withdrawal and the other variables 
are probably underestimates of their true magnitude for two reasons. First, there was decided re- 
striction of range on the withdrawal measure that leads to a ceiling on the possible size of the 
correlations that may be obtained. Second, there was probably some sample selection bias be- 
cause some unknown number of teachers who felt adversely affected by the Benchmark Goals 
Program in earlier time periods may have already left the district or had already transferred to a 
more satisfying school within the district. 
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Table 11. Effect of Compensation Variables on Teacher Turnover in Charlotte-Mecklenburg (N=1,15Q) 



Total Turnover Leave the District Transfer Between 

Schoois 



Variable 


Standardized 


Odds 


Standardized 


Odds 


Standardized 


Odds 




Coefficient 


Ratio*®' 


Coefficient 


Ratio^®^ 


Coefficient 


Ratio^®^ 


Yrs in Current Job 


-0.13** 


0.785 


-0.05 


0.914 


-0.17** 


0.738 


MA&MSor More (0/1) 


0.00 


1.004 


-0.05 


0.824 


0.04 


1.159 


Age 


-0.01 


0.999 


-0.15* 


0.765 


0.09 


1.188 


Teach Science or Math 
(0/1) 


0.07 


1.567 


-0.04 


0.769 


0.11*** 


2.048 


High School (0/1) 


0.06 


1.463 


-0.07 


0.619 


0.13** 


2.257 


Middle School (0/1) 


0.06 


1.358 


0.04 


1.186 


0.08 


1.428 


Gender (0/1, F=1) 


-0.01 


0.957 


-0.03 


0.823 


0.01 


1.087 


Minority (0/1) 


0.00 


0.985 


0.00 


1.018 


0.00 


0.968 


Percent Free/ 
Reduced Price Lunch 


0.12*** 


1.237 


0.02 


1.033 


0.17*** 


1.354 


Pay Level 


-0.12* 


0.806 


-0.13 


0.788 


-0.10 


0.837 


Pay Level 
Satisfaction 


0.09** 


1.171 


0.13** 


1.266 


0.03 


1.058 


Teach in Bonus School 
(1=Yes) 


-0.08* 


0.755 


0.03 


1.112 


-0.14*** 


0.599 


Fair to Receive 
Bonus 


-0.03 


0.951 


-0.01 


1.002 


-0.05 


0.911 


Intent to Leave 


-0.24*** 


1.804 


0.29*** 


1.691 


0.17*** 


1.350 



Pseudo JOO ^93 ^82 

* significant at the AO level ** significant at the .05 level *** significant at the .01 level 



(a) For 0/1 variables, the odds ratio is the change in the odds of leaving the district or transferring associated with the variable 
taking on a value of 1. For the others, it is the change in odds associated with a one standard deviation increase in the value of 
the variable. An odds ratio greater than 1 indicates a increased chance of leaving the district or transferring. An odds ratio less 
than 1 indicates an decreased chance of leaving the district or transferring. 



We offer the following interpretation of these results (also discussed in Heneman and Mi- 
lanowski, 1998). Pay level appeared to influence both leaving the district and transferring be- 
tween schools, whereas being in a bonus school influenced only transferring. This suggests that 
teacher withdrawal generally is more heavily influenced by base pay, which makes up 96 percent 
or more of their total pay, than by the bonus or other rewards associated with working in a bo- 
nus-winning school. Subjective reactions to pay level, as well as objective pay level, is important 
to account for teachers’ withdrawal responses, since satisfaction with pay level was also signifi- 
cantly related to leaving the district. However, the relationship was unexpectedly positive, so that 
even after controlling for the numerous factors that might covary with either variable, teachers 
more satisfied with their pay level were more likely to leave the district. This puzzling result 
might be due to a subgroup of teachers who used a noncompensatory model to evaluate and re- 
spond to the totality of their terms and conditions of employment. That is, these teachers may 
have been satisfied with their base pay, but not other attributes of their job, and the pay could not 
or did not compensate for these other attributes, so they left the district. 



Support for this interpretation comes from responses of Charlotte-Mecklenburg teachers in the 
interviews previously described. In response to questions about the Benchmark Goals Program 
and their possible bonus, many teachers spoke in very harsh terms about these matters (Hene- 
man, 1998). While teachers generally valued the focus and challenge of the student achievement 
goals, many felt extreme pressure to achieve the goals and reported that they were working more 
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hours (during the day or on weekends) in order to change what and how they taught in order to 
reach the goals. These teachers frequently mentioned increased stress when asked to describe 
how their jobs had changed for the worse under the Benchmark Goals Program. In short, these 
teachers did not view the new effort/pay level ‘bargain’ as a ‘good deal,’ regardless of whether 
or not they were working in an award winning school, and thus they may have chosen to leave 
the district. 

This interpretation is consistent with the unfolding model of turnover proposed by Lee and 
Mitchell (1994). They hypothesize that some turnover is due to the employee experiencing a 
shock (such as the Benchmark Goals Program), which leads to one of four distinct quitting proc- 
esses or paths. The paths differ in large part according to whether the employees quit with exter- 
nal job alternatives known (a pull process) or without job alternatives known (a push process). 
Based in part on the unfavorable comments about the Benchmark Goals Program made by some 
teachers in the interviews, we speculate that the teachers who left the district and were relatively 
satisfied with their base pay were likely on a push path. Additionally, leaving the district to teach 
in another district would require finding a job outside of the county in a smaller district with 
fewer job alternatives, accepting base pay likely to be the same or less than current base pay, and 
being willing to relocate to take the job. These factors mitigate against a strong external pull ex- 
planation. Finally, the teachers most likely to be marketable and pulled away from teaching alto- 
gether are those with advanced degrees and teaching math or science subjects, but the coeffi- 
cients for these control variables were not significant, again suggesting little labor market pull on 
teachers to leave the district. 

Teaching in a school that had received bonuses did not influence a decision to leave the district, 
but it did reduce the probability of transfer, implying that teachers tend to transfer out of non- 
award schools (presumably to an award-winning schools, though we could not determine that 
from the data base). This finding would be explained by a lack of external job alternatives, a de- 
sire to remain in teaching, and wanting to work in the more positive work environment more 
likely to be found in award-winning schools. Such a positive work environment is probably 
made up of a constellation of both extrinsically rewarding job attributes (for example, receiving 
the bonus, pride in achieving goals, and public recognition) and intrinsically rewarding attributes 
(such as autonomy and coworker supportiveness). At least for some teachers, escaping non- 
award-winning schools was important, though it is not possible to say how much of this was due 
to not receiving the bonus or other positive outcomes and how much was due to other working 
conditions that might be associated with not achieving award status. To control such withdrawal, 
it would be advisable for a district using school-based performance awards to take steps to im- 
prove schools’ chances of being award winners. 

The perceived fairness of the amount of the bonus that could be won had no impact on with- 
drawal. Perhaps this reflects the relatively small size of the bonus, so that the bonus was not a 
“noticeable difference” to teachers relative to other job attributes and thus was not a reward that 
by itself would cause withdrawal, regardless of how fair it was perceived to be. It is possible that 
perceived bonus fairness would have had a larger impact on withdrawal decisions had the bonus 
been larger. 
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The most important predictor of both leaving the district and transferring to other schools in our 
models was intent to leave in reaction to experiences with the Benchmark Goals Program. The 
multiple outcomes (besides bonus) associated with the Benchmark Goals Program, and teachers’ 
previously noted reactions to these outcomes, undoubtedly carried over into the formation of 
withdrawal intentions. In short, pay level satisfaction, being in a school that did not win an 
award, and intent to leave because of experiences with the awards program all played a role in 
teachers’ decisions to leave the district or transfer between schools within the district. 

Teacher Perceptions of Program Fairness 

In the interviews, teachers often identified program design and context factors that they felt made 
the awards program less fair (see Heneman, 1998; Kelley, 1998a; Kelley and Protsik, 1997). 
Teachers and principals in some schools said that it was unfair to put so much pressure on the 
teachers in the assessment subjects and grade levels, although this became less of an issue in 
Kentucky when the state split the assessment across six instead of three grade levels. Teachers in 
some Kentucky schools did not clearly understand how school goals were set, and some lacked 
confidence in the validity and reliability of the assessment instrument. In Charlotte- 
Mecklenburg, teachers in a magnet school expressed concern that the goals of the school-based 
performance award program differed from the goals of the magnet program. Although teachers 
generally agreed that it was fair to hold them accountable for student performance (Heneman, 
1998), some felt that differences across schools made it harder or easier to improve performance. 
Teachers said that high student mobility, high at-risk student populations (due to learning dis- 
abilities or low socioeconomic status of students) made it difficult for them to improve student 
performance (Kelley and Protsik, 1997). Teachers in the higher grade levels sometimes ex- 
pressed concern about low student motivation to perform well on the assessment. Teachers in the 
most successful schools were somewhat less likely to mention fairness as a problem. Teachers 
and principals in these schools acknowledged issues such as mobility and student motivation, but 
they tended to mention actions taken to address these issues, rather than to dwell on them as in- 
surmountable barriers to meeting their schools’ goals. Principal leadership seemed to be critical 
in shaping and addressing teacher concerns about fairness (Kelley, 1998a). 

By including in our survey a set of items dealing with the fairness of various aspects of school- 
based performance award programs, we were able to follow up on the fairness issues raised in 
the interviews. Table 12 presents the means and standard deviations for the teacher responses to 
these items. (Appendix Table AlO shows the items combined to form scales and the internal con- 
sistency reliabilities.) 

While Charlotte-Mecklenburg teachers, on the average, expressed weak agreement that it is fair 
to hold teachers accountable for student achievement and fair for teachers who increase student 
achievement to receive a bonus, Kentucky teachers on the average disagreed with both ideas. 
Teachers in both jurisdictions were just below the neutral point as to whether they felt their 
school had a fair chance of winning the award. Teachers in both jurisdictions disagreed, on aver- 
age, that the program was run in a fair way. As might be expected, teachers in schools that had 
consistently won awards had more positive perceptions of fairness than teachers in schools that 
had never won or had won only once, like in Charlotte-Mecklenburg. However, the differences 
in means were only about .2 to .4 scale points, and even for teachers in schools that had been 
consistent winners, the averages for the program fairness scale were still below the neutral point 
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in both jurisdictions. This suggests that many teachers had reservations about the fairness of the 
way the programs were run. Since we found evidence, discussed earlier, that perceptions of fair- 
ness are important determinants of teacher expectancy, these perceptions of unfairness may have 
limited the motivational impact of the programs we studied. 



Table 12. Teachers' Perceptions About the Fairness of the School-Based Performance Award Programs 



Item 


Charlotte 

Mean^ 


Charlotte 

Standard 

Deviation 


Kentucky 

Mean^ 


Kentucky 

Standard 

Deviation 


Fair to hold teachers accountable for 
student achievement. 


3.3 


1.1 


2.5 


1.2 


Fair for teachers who increase student 
achievement to get a bonus. 


3.6 


1.1 


2.8 


1.2 


School has a fair chance to win award 
(2 item scale). 


2.8 


1.2 


2.9 


1.1 


Program is procedurally fair 


2.6 


0.9 


2.4 


0.9 



(3 item scale). 

(a) Response scale ws I =Strongly Disagree . . .5=Strongly Agree. 

All differences between comparable Charlotte-Mecklenburg and Kentucky averages are statistically significant at the .05 level or 
beyond. 



DOES TEACHER MOTIVATION AFFECT 
SCHOOL PERFORMANCE? 



In addition to looking at how school-based performance award programs affect individual 
teacher’s motivation, it is important to explore whether teachers’ motivation affects subsequent 
school performance, as defined by the particular award program. To investigate this, we aver- 
aged the teachers’ motivational responses assessed during the accountability cycle within each 
school, then used these school averages to predict subsequent school performance on the ac- 
countability indices at the end of the cycle. Based on our motivational model, we expected that 
the greater the average teacher motivation, the greater the school’s success in improving student 
achievement levels. 

Variables 

We calculated school-level averages of teachers’ individual expectancy perceptions and the ex- 
pected value of outcomes (desirability times probability of occurrence) in order to represent the 
average level of teacher motivation at each school. The expected value of the bonus alone was 
calculated, along with a summed expected value for each of the four outcome groupings pre- 
sented in Table 5 on page 16: goal attainment, learning, sanctions, and stress. Individual scores 
were then averaged across teachers within each school. 

Whenever individual-level data are aggregated to form a score representing a higher-level entity 
such as a school, we need to be concerned about the degree of agreement in the individual re- 
sponses that are averaged. When using the within-school averages of teachers’ ratings of their 
motivational responses to represent the average level of motivation within each school, within- 
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school variability in individual teachers’ responses leads to sampling error in the estimate of each 
school average. This is a form of unreliability that is likely to have attenuated the relationships 
between the school-level averages of the motivational constructs and the measures of school per- 
formance. The issue of within-school variability is discussed further in the Technical Appendix. 

We defined school performance using the same measures each program used to provide rewards 
and sanctions to teachers and schools. For Charlotte-Mecklenburg schools, performance was de- 
fined by the number of Benchmark Goals points each school achieved in the 1996-97 school 
year, as reported by the district. For Kentucky, the indicator was the change in the KIRIS index 
between the baseline (essentially the KIRIS index from Cycle 2) and the index for Cycle 3 (con- 
sisting of the average of the 1997 and 1998 indices). We obtained these data from the Kentucky 
Department of Education. 

It should be noted that in neither jurisdiction was the school performance measure a simple im- 
provement in test scores. These performance measures were indices that summarize performance 
on several dimensions and represented the particular jurisdictions’ definition of school perform- 
ance, rather than being a standardized measure of student achievement. Charlotte-Mecklenburg 
schools earned Benchmark Goals points not only by improving average scores on a wide variety 
of state and district assessments but also by maintaining desired levels already achieved. For a 
typical elementary school, approximately 62 percent of the maximum possible points could be 
earned by improving test scores, 36 percent by improving primary grade readiness assessments, 
and 2 percent by reducing absenteeism. For a typical high school, 60 percent of the points were 
related to improvement in test scores, 29 percent to enrollment in higher level classes, and 1 1 
percent to reduction of absenteeism and dropout rates. Similarly, Kentucky’s KIRIS index was 
based on test scores in several subjects (actually on the percentage of students scoring within cer- 
tain proficiency ranges on tests) and on indicators like dropout and retention rates. In calculating 
the KIRIS index, improvements in student academic proficiency were given approximately 84 
percent of the weight while the other indicators were given approximately 16 percent. The state 
administered rewards and sanctions based on changes in the KIRIS index. Because of the unique 
features and complex construction of each index, each represents a more complex concept of 
school performance than, for example, scores on the National Assessment of Educational Pro- 
gress tests do. In our research, we have accepted each jurisdiction’s definition of school perform- 
ance. The school performance measures are best thought of as indicating the degree to which 
schools attained the goals the award programs set for them. 

In the following analyses, we controlled for certain school characteristics that might affect how 
well schools did on the accountability indices. These included school size (represented by the 
natural log of enrollment), and the percentage of students receiving free and reduced-price lunch. 
For the Kentucky analysis, we included the value of the Cycle 2 KIRIS index because it might 
have been easier for schools with relatively low levels on the Cycle 2 index to improve enough 
to be a reward school than it was for schools with higher levels on the index. In Charlotte- 
Mecklenburg, many of the improvement goals underlying the Benchmark Goals points were ad- 
justed for this possibility, so previous school performance was not used as a control variable. 

Data for these control variables were provided by the Charlotte-Mecklenburg district and the 
Kentucky Department of Education. 
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Prediction of Schooi Performance 

We used regression analyses to estimate the effect of teacher motivation (as represented by ex- 
pectancy and the expected value of outcomes) on school performance, controlling for the school 
characteristics mentioned above. The results are presented in Table 13. (Means, standard devia- 
tions, and intercorrelations of the variables are shown in Technical Appendix Table A12.) We 
estimated six regression models, along with a controls-only model, to reflect the six different 
treatments of expected value of outcomes (bonus only, goal attainment only, learning outcomes 
only, sanctions only, stress only, and all types together). These separate analyses allowed us to 
probe for potentially different effects of each type of outcome. Table 13 presents only results of 
additive models, since adding multiplicative terms (expectancy times expected value) did not 
significantly increase variance explained. 



Table 13. The Effect of Teacher Expectancy and Expected Outcome Value on School Performance 
(N=114 Charlotte-Mecklenburg Schools and 204 Kentucky Schools) 



Charlotte-Mecklenburg 


Con- 

trols 

Only 


Value of 
Bonus 


Goal 

Attain- 

ment 


Learning 


Sanctions 


Stress 


All 4 
Types of 
Outcomes 


Log Enrollment 


-.310*** 


-.260*** 


-.252*** 


-.264*** 


-.276*** 


-.273*** 


-.293*** 


Percent Free/Reduced 


-.542*** 


-.527*** 


-.520*** 


-.487*** 


-.512*** 


-.536*** 


-.486*** 


Lunch 
















Expectancy 




.239*** 


.288*** 


.343*** 


.269*** 


.274*** 


.343*** 


Expected Value of Bonus 




.049 












Goal Attainment Outcomes 






-.050 








.048 


Learning Outcomes 








-.170* 






-.179 


Sanctions 










.122 




.122 


Stress Outcomes 












-.142* 


-.154* 




.216 


.284 


.284 


.302 


.296 


.302 


.337 


Kentucky 


Con- 


Value of 


Goal 


Learning 


Sanctions 


Stress 


All 4 




trols 


Bonus 


Attain- 








Types of 




Only 




ment 








Outcomes 


Log Enrollment 


-.143** 


-.129** 


-.142** 


-.145** 


-.134*** 


-.145** 


-.129* 


Percent Free/Reduced 


-.484*** 


-.496*** 


-.512*** 


-.523*** 


-.517*** 


-.519*** 


-.514*** 


Lunch 
















Cycle 2 KIRIS Index 


-.601*** 


-.710*** 


-.732*** 


-.740*** 


-.680*** 


-.728*** 


-.727*** 


Expectancy-Reward Status 




.290*** 


.283*** 


.251*** 




.266*** 


.298*** 


Expectancy-Avoid Decline 










.222*** 






Expected Value of Bonus 




-.137** 












Goal Attainment Outcomes 






-.087 








-.140 


Learning Outcomes 








-.015 






.098 


Sanctions 










-.068 




-.082 


Stress Outcomes 












-.088 


-.040 


R^ 


.283 


.342 


.333 


.328 


.326 


.334 


.345 



*** Statistically significant at the .01 level 
** Statistically significant at the .05 level 
* Statistically significant at the . 1 0 level 



While the controls-only model accounted for significant variance in school performance, adding 
the variables representing teacher motivation significantly increased the proportion of variance 
explained, indicating that teacher motivation helps predict school performance. In all models, 
average expectancy had a positive coefficient of substantial magnitude and was significant at the 
.01 level. Thus, the higher the average teacher expectancy, the more Benchmark Goals points a 
Charlotte-Mecklenburg school achieved or the greater the increase in the KIRIS index a Ken- 
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tucky school achieved. In the Charlotte-Mecklenburg analyses, a one standard deviation increase 
in average teacher expectancy (where one standard deviation equals 12.88 percentage points on 
the 0 to 100 percent likelihood scale) was associated with an increase in Benchmark Goal points 
of .239 to .343 standard deviations, depending on which outcome expected values were also in- 
cluded in the analysis. Since the standard deviation of the Benchmark Goal points was 15.73, 
the expected effect of a one standard deviation increase in average teacher expectancy was a 3.8 
to 5.4 point increase in Benchmark Goal points. In the Kentucky analyses, a one standard devia- 
tion increase in average teacher expectancy of achieving reward status (where a standard devia- 
tion equals 14.35 percentage points on the 0 to 100 percent likelihood scale) was associated with 
an increase of .251 to .298 standard deviations on the KIRIS index. Since one standard deviation 
of the KIRIS index improvements equals 4.70 points, the expected effect of a one standard 
deviation increase in teacher expectancy was between 1.2 and 1.4 points of improvement on the 
KIRIS index. A one standard deviation increase in average teacher expectancy of avoiding a de- 
cline in the KIRIS index, which could lead to sanctions (where a standard deviation equals 12.20 
percentage points on the 0 to 100 percent likelihood scale) is associated with an increase of .222 
standard deviations on the KIRIS index. The effect of a one standard deviation increase in 
teacher expectancy of avoiding declines in the KIRIS index is a one point improvement on the 
KIRIS index. 

While teacher expectancy had the expected positive effect on school performance, the expected 
value of outcome variables have relatively small coefficients, and some have unexpected signs. 
We anticipated that schools at which teachers had on the average higher expected values for the 
bonus, goal attainment, and learning outcomes would perform better, controlling for expectancy, 
because they would be more highly motivated by the chance to receive these positive outcomes. 
Our expectations were not realized. The expected values for these outcomes had either insignifi- 
cant or significant but negative coefficients. These analyses therefore do not provide any evi- 
dence that schools at which the positive outcomes were more highly valued, and seen as more 
likely to be received if goals are met, did better on the school performance measure. 

The coefficients for the sanctions outcomes were small and not statistically significant. This re- 
sult does not support the expectation that schools will perform better in which teachers, on the 
average, value these outcomes more negatively and expect them to occur if the school fails to 
perform, implying that these sanctions do not have a strong motivational value. Finally, with re- 
spect to the stress outcomes, the coefficients are again small, and only statistically significant for 
the Charlotte-Mecklenburg schools. The negative signs are consistent with the expectation that 
schools in which teachers negatively valued these experiences and saw them as a more likely 
consequence of trying to meet the goals would not perform as well. 

Similar analyses using only elementary schools (n=81 for Charlotte-Mecklenburg, n=83 for Ken- 
tucky) did not show any substantially different results from those reported above. 

Overall, these analyses strongly suggest that between-school differences in expectancy were re- 
lated to the success of schools in improving student performance as defined by the school-based 
performance award programs in these two sites. However, between-school differences in the de- 
gree to which teachers valued and expected to receive the various outcomes, including bonuses, 
associated with the goal achievement appear to be mostly unrelated to differences in school goal 
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achievement. These latter results were contrary to our theoretical expectations. As explained 
later, these results may be due to problems of unreliability in measurement. 



PRINCIPALS' REACTIONS TO SCHOOL- 
BASED PERFORMANCE AWARD 
PROGRAMS 



In this section, we examine the perceptions of principals based on surveys administered in Char- 
lotte-Mecklenburg, Kentucky, and Maryland. Principals in the three jurisdictions were asked 
similar questions on surveys administered to them by mail in 1997 (Charlotte-Mecklenburg and 
Kentucky) and 1998 (Maryland). We first compared the reactions of principals to the reactions of 
teachers in Charlotte-Mecklenburg and Kentucky and then compared principal responses across 
the three Jurisdictions. These data provided us with some clues about how variations in program 
design and program context produce different perceptions at the school level. We use the data to 
hypothesize ways in which different design features might produce different results. 

Research evidence and common sense suggest that program design and program context are im- 
portant determinants of program effects (Bennett, 1992; Bullock and Tubbs, 1991; Odden and 
Kelley, 1997). These three performance award programs provided interesting contrasts in level 
and focus of award, in presence or absence and focus of sanctions, and in differences of context. 
Some potentially important distinctions in program design and program context among the three 
programs are as follows. Both the Charlotte-Mecklenburg and Kentucky programs provided sal- 
ary bonuses to teachers, while the Maryland program paid awards to schools in the form of 
school improvement funds. The Charlotte-Mecklenburg program provided lower awards more 
frequently ($750 to $1,000 per year), while the Kentucky program paid larger awards (roughly 
twice the Charlotte-Mecklenburg award) at the end of a two-year award cycle. In Charlotte- 
Mecklenburg, teacher salaries were relatively low compared to other states, while Kentucky in- 
fused additional resources into the education system to raise teacher salaries. Maryland made 
awards to a smaller proportion of schools than in Charlotte-Mecklenburg or Kentucky (10 per- 
cent versus 40 percent). The Charlotte-Mecklenburg program had no formal sanctions compo- 
nent, although the district placed significant pressure on principals to improve school perform- 
ance toward the accountability goals. The Kentucky program also had a strong sanctions compo- 
nent that put all schools at risk of sanctions if their performance declined, regardless of absolute 
level of performance. Therefore, Kentucky’s sanctions could affect the highest as well as the 
lowest performing schools in the state. The Maryland program includes sanctions, but they have 
not been widely used and were seen as applying only to Baltimore city schools. 

The Charlotte-Mecklenburg program was created in part to address concerns about disparities in 
achievement between white and African-American students. The district’s accountability goals 
attempted to raise the performance of all students, with a particular emphasis on improving the 
performance of the lowest-performing students and minority students. The Kentucky program 
was part of a massive court-ordered overhaul of the state’s entire education system, including a 
reduction in local community control, a direct state-to-school reform effort, and comprehensive 
initiatives including mandatory site-based management, ungraded primary schools, and signifi- 
cant investment in technology, professional development, tutoring, and family services. Ken- 
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tucky’s accountability goals attempted to motivate significantly higher levels of performance and 
a shift to new models of instruction that emphasized performance, relationships to real-world 
problems and higher-order thinking skills. Maryland has been a leader in standards-based reform, 
and added the performance award for elementary and middle schools after the assessment was in 
place. Because Maryland has only 24 districts, the Department of Education can work closely 
with districts to implement the accountability program. 

Principal and Teacher Reactions in Charlotte-Mecklenburg and Kentucky 

Table 14 summarizes the mean responses of principals in Charlotte-Mecklenburg and Kentucky 
to questions about goal clarity, goal conflict, and goal commitment, and it compares principal 
with teacher means. Principals exceeded teachers in all three areas. From an accountability 
standpoint, Charlotte-Mecklenburg principals provided the most desirable responses. Of the four 
groups, they were highest on clarity and commitment and lowest on goal conflict. This is perhaps 
not surprising, because the district had in place a strategy of hiring principals who were commit- 
ted to the accountability goals and of replacing those whose schools failed to meet accountability 
expectations. 



Table 14. Mean Responses of Charlotte-Mecklenburg and Kentucky Principals and Teachers to Goal- 

Related Scales 



Scale 


Charlotte* 

Mecklenburg 

Principals 


Charlotte* 

Mecklenburg 

Teachers 


Kentucky 

Principals 


Kentucky 

Teachers 


Goal Clarity 


4.2 


3.4 


4.1 


3.6 


Goal Conflict 


2.6 


2.8 


2.9 


3.2 


Goal Commitment 


4.0 


3.9 


3.7 


3.9 



Differences between principal and teacher averages within each jurisdiction are statistically significant at the . 05 level or be- 
yond: differences in principal and teacher averages across Jurisdictions are not statistically significant. 



A Comparison of Principal Responses Across Programs 

Table 1 5 presents the mean responses of principals to questions about the rewards provided by 
the programs. Again, Charlotte-Mecklenburg principals, on average, had the most positive per- 
ceptions of rewards as motivators, followed by Maryland principals. Kentucky principals were 
least enthusiastic about the rewards. 

It is interesting that both Maryland and Charlotte-Mecklenburg principals were more optimistic 
about the motivational potential of rewards than were Kentucky principals. Charlotte- 
Mecklenburg and Kentucky rewarded with pay bonuses while Maryland used school improve- 
ment funds, so apparently, the form of the reward does not account for these differences in prin- 
cipals’ evaluation of motivational impact. Kentucky principals may have been relatively negative 
about performance rewards because on average, they did not view the accountability system as a 
fair way to measure student performance. Another possible explanation is that Kentucky princi- 
pals on average estimated a lesser likelihood of achieving reward-level performance (if the prin- 
cipal and staff worked together as hard as they could): 58 percent for Kentucky; 70 percent for 
Maryland; and 71 percent for Charlotte-Mecklenburg. 

Table 15. Mean Responses^ of Principals to Selected Reward Items 

Item Charlotte Kentucky Maryland 

Principals Principals Principals 
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1 . Fair for teachers/schools that improve student achievement 
to receive reward 


4.4 


2.9 


3.7 


2. Reward helps me motivate teachers 


3.4 


2.3 


3.0 


3. Reward provides shared goals 


3.7 


2.4 


3.2 


4. Tripling size of reward would improve teacher motivation 


4.2 


2.3 


2.9 


5. Accountability system is a fair way to measure school 
performance 


- 


2.5 


3.0 


6. Award part of accountability program should be continued 


- 


2.3 


3.5 


7. Paying bonus directly to teachers would more 
effectively motivate them 


- 


- 


3.5 



(a) Response scale: /= Strongly Disagree. ..5 - Strongly Agree 

All differences are statistically significant at the .05 level or beyond except between Charlotte-Mecklenburg and Maryland prin- 
cipals for items 2 and 3, and between Kentucky and Maryland principals on item 4. 



To summarize the entire set of principal responses and allow a more parsimonious comparison 
across programs, we factor analyzed common survey items from the Kentucky and Maryland 
questionnaires. (The Charlotte-Mecklenburg questionnaire did not include as many common 
items.) The survey items used one of two five-point rating scales, in which (depending on the 
question) 1 = not at all or strongly disagree; 3 = to a moderate extent or neither agree nor dis- 
agree, and 5 = to a very great extent or strongly agree. We identified seven factors: the impor- 
tance of financial incentives in motivating teacher performance; district support for program 
goals principal understanding and commitment of program goals; the allocation of resources to 
support the goals of the program; principal skills in helping teachers align and improve teaching 
practice; and teacher teamwork and learning in support of goal achievement (Kelley, Conley and 
Kimball, 1999). Technical Appendix Tables A13, A14, and A15 provide a breakdown of the 
items comprising each factor, reliabilities, and factor loadings. We created scale scores by aver- 
aging the Kentucky and Maryland items that loaded most strongly on each factor. We created 
Charlotte-Mecklenburg scale scores by averaging the corresponding items. Table 16 compares 
responses from Charlotte-Mecklenburg, Kentucky, and Maryland for each of the seven factors. 



Table 16. A Comparison of Mean Principal Responses Across Performance Award Programs in 



Charlotte-Mecklenburg, Kentucky, and Maryland 



Scale 


Charlotte-Mecklenburg 
(Salary Bonus, 
Principal Sanctions) 


Kentucky 
(Salary Bonus, 
Sanctions) 


Maryland 
(Award to School, 
Limited Sanctions) 


Financial Incentive 


- 


2.36 


O.IO'^ 


Improvement of Teaching 


3.73 


3.82'^ 


3.67 


Principal Understanding 


4.28'^ 


4.13 


3.82 


District Support 


2.92 


3.69'^ 


3.61 


Resource Allocation 


3.49 


3.51'^ 


3.15 


Principal Skill 


3.93^^ 


3.69 


3.80 


Teacher Teamwork and 
Learning 


3.90^^ 


3.80 


3.87 



Indicates highest mean in the triad. 

All differences between the lowest and highest pair in the triad are statistically significant at the .05 level or beyond. 



Because the Charlotte-Mecklenburg survey did not include a sufficient number of similar items 
related to financial incentives, no Charlotte-Mecklenburg factor score was calculated for the fi- 
nancial incentives group. For the other factors, Charlotte-Mecklenburg was highest of the three 
programs on principal understanding, principal skill, and teacher teamwork and learning. Princi- 
pal understanding and principal skill may have been highest in Charlotte-Mecklenburg because 
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the program included easily assessed structural changes (such as increased enrollment in higher- 
level courses) and improvement in achievement that did not generally require wholesale modifi- 
cations of curriculum and instruction (as did the Kentucky and Maryland programs). The teacher 
teamwork and learning factor was highest in Charlotte-Mecklenburg, although the differences 
across the three programs were not substantively significant. Examination of the individual items 
comprising this factor shows that Kentucky is highest on opportunities for professional develop- 
ment related to the assessment and on opportunities to learn best practices from other schools. 
Differences in these items probably reflect Kentucky’s initiatives in these areas. This factor also 
included two other items related to structural changes that provided common planning time to 
teachers to work together on improving teaching practice. Both Charlotte-Mecklenburg and 
Maryland were high on these items, which may reflect efforts in both jurisdictions to evaluate 
principals based on the accountability goals. 

Improvements in teaching and learning may be the most desirable outcome of a school-based 
performance award program. Table 16 shows that Kentucky’s program, with salary bonuses and 
sanctions operating in the context of a broad-scale education reform effort, was highest on the 
improvement of teaching, district support, and resource allocation. This suggests that, of the 
three program types, award programs with sanctions in the context of a larger reform effort have 
the greatest impact on improvement of teaching, district support, and resource alignment. Princi- 
pals in the two programs with salary bonuses rated improvement in teaching, principal under- 
standing, and resource allocation significantly higher than did principals in Maryland’s program, 
which made awards in the form of school improvement funds. 

These data are not conclusive, and implications regarding program design should be considered 
in the larger cultural, educational, and reform context. Still, one interpretation of these findings is 
that a school-based performance award program with sanctions provides the most powerful ef- 
fect, enhancing principal understanding and resource allocation, and motivating the greatest level 
of effort to modify teaching practice to achieve accountability goals. This type of program ap- 
pears to provide the most powerful jump-start for the system, but also creates the greatest nega- 
tive reaction from principals and teachers. Based on principals’ reported perceptions, all three 
programs promoted some changes in teaching practice, teacher teamwork and learning, and at 
least some modest reallocation of resources to support reform. Policymakers will need to assess 
the need for change and weigh the trade-offs. 



CONCLUSION 



Our study of school-based performance award programs suggests that they are a potentially use- 
ful tool for policymakers and administrators interested in focusing teachers, principals, and edu- 
cational systems on improving student performance. Their full potential may be difficult to 
reach, however, due to the complexity of designing and implementing such programs. We found 
that the programs we studied had mixed success in fulfilling the conditions that our motivational 
model suggests are important. Teachers in the programs we studied reported that they understood 
the program goals and were committed to them. The programs appeared to have a focusing effect 
on teaching practice, collaboration and teamwork, and resource alignment. Teachers who per- 
ceived the bonus and other positive outcomes associated with achieving the goals as having a 
higher expected value reported more understanding of and commitment to the goals. Though 
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most teachers reported trying to reach the goals, many were skeptical about continuing to reach 
ever-rising goals. This skepticism was higher in schools that had previously met their goals. 

Teachers perceived that both desirable and undesirable outcomes were associated with the award 
programs. The most desirable outcomes included personal satisfaction, seeing students learn new 
skills, having clear goals, and receiving a bonus. Least desirable outcomes included public criti- 
cism or embarrassment, additional pressure and job stress, and risk to job security. The outcomes 
may be grouped into four clusters: goal attainment (bonus, funds for school improvement, per- 
sonal satisfaction from meeting the goals, public recognition); learning (having clear goals, pro- 
fessional development opportunities, working cooperatively with other teachers, having students 
leam new skills, satisfaction that student performance increased); stress (more pressure and job 
stress, working more hours, less freedom to teach things unrelated to the goals); and sanctions 
(loss of pride, public criticism, loss of job security, intervention). Outcomes in the first two clus- 
ters received high desirability ratings and outcomes in the last two clusters received low desir- 
ability ratings. Outcomes such as personal satisfaction from achieving the goals were as impor- 
tant to teachers as the formal rewards of the programs such as receiving a bonus, and teachers 
who rated goals like having students leam new skills as highly desirable did not rate receiving a 
bonus as highly undesirable. Bonuses per se do not seem to be seen by these teachers as anti- 
thetical to the intrinsic motivators of teaching. 

While the desirability of receiving a bonus was rated relatively high, teachers were less than cer- 
tain that achieving the goals would lead to receiving the bonus. (The average probability per- 
ceived by Charlotte-Mecklenburg teachers was 72.8 percent; by Kentucky teachers, 54.8 per- 
cent.) Teachers in both jurisdictions perceived a greater likelihood of experiencing personal sat- 
isfaction, but also of experiencing more stress, and in Kentucky, of longer hours and less free- 
dom to teach. This suggests that, at least in Kentucky, the attraction of receiving a bonus may 
have been outweighed by the negative aspects of trying to meet the goals, thus weakening the 
potential motivating power of the bonus and other positive outcomes. 

Teachers’ expectancy perceptions (beliefs that their efforts would result in their school’s meeting 
its goals) were weaker than expected. The individual expectancy was, on the average, quite low 
(39.2 percent in Kentucky and 55.6 percent in Charlotte-Mecklenburg). The average group ex- 
pectancy perceptions were somewhat higher. Teachers reported higher levels of individual and 
group expectancy in schools in which teachers perceived stronger assessment data feedback 
mechanisms, principal support or professional community, lack of goal conflict, and higher lev- 
els of program fairness. Past performance in the program was important as well: teachers in 
schools with a history of success with the program reported higher levels of expectancy. 

Kentucky teachers found the bonus per se desirable but they were not, on the average, highly in 
favor of having the bonus program continue, nor did they report that they were highly motivated 
by the bonus. Teachers who said they were more motivated by the bonus and wanted to see it 
continue were more dissatisfied with their current salary, felt that the use of bonuses and the bo- 
nus process were fair, and felt that teachers should decide how to divide up the award money. 
Teachers who reported a greater desire to withdraw from the bonus program were those who saw 
the program as less fair. 
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We found that schools with higher average levels of teacher expectancy were more likely to 
show improvements in student achievement. This is consistent with the part of our motivational 
model that suggests that success is more likely to call forth higher levels of effort. These higher 
levels of effort in turn lead to higher levels of performance. This is consistent with results of re- 
search associating expectancy beliefs with performance. (See Van Eerde and Thierry, 1996, for a 
meta-analysis, and Tschannen-Moran, Hoy, and Hoy, 1998, for a discussion of research on a re- 
lated construct, teacher efficacy.) The similarity of the magnitude and direction of the effect of 
average teacher expectancy on school performance across the two jurisdictions is notable, even 
though the actual measures of school performance (that is, the way student achievement was 
measured and combined with other indicators) and other program design features were quite dif- 
ferent. This cross-jurisdiction similarity provides some evidence of the generalizability of the 
teacher expectancy-school performance relationship. 

We had expected that greater value placed on the outcomes coupled with greater certainty that 
performance would lead to positive outcomes would lead to more effort, which would lead in 
turn to higher performance. But contrary to our expectations, the schools for which the average 
expected value of the positive outcomes (the bonus by itself or the goal attainment outcomes as a 
group) was higher were not more likely to show greater improvements in school performance. 
Nor were schools for which the average expected negative value of sanctions was greater more 
likely to show greater improvements. This finding that higher expected values of outcomes was 
not associated with higher performance is inconsistent with a substantial body of previous re- 
search (see Van Eerde and Thierry, 1996). We do not interpret these results to mean that teach- 
ers’ valuation and beliefs about the likelihood of program rewards and sanctions do not influence 
teacher motivation. Rather, we attribute the results to a combination of measurement error, limi- 
tations in our research design (see the Strengths and Limitations of Study Methods section be- 
low) and, in the case of the bonus, the relatively weak nature of the intervention. 

Comparing the perceptions of principals in Charlotte-Mecklenburg and Kentucky (both salary 
bonus programs) with the perceptions of principals in Maryland (a program that provided school 
improvement funds), we found that program design and program context appear to be important 
factors affecting motivation. Among the three programs, principals in Kentucky reported the 
greatest change in teaching practice. The Kentucky program had the largest salary bonus and the 
strongest sanctions component. In addition, principal understanding of, and resource allocation 
toward, the goals of the program were higher in the salary bonus programs. Both survey and in- 
terview responses from Kentucky principals suggested that the bonus is not a strong motivator. 
One interpretation of these findings is that salary bonus programs with sanctions focus teachers 
and educational systems on the improvement goals, but also produce pressure and stress, result- 
ing in negative attitudes about the program and its effectiveness. Changes to teaching practice 
appear to come at a price. 

Implications for the Design of School-Based Performance Award Programs 

That the motivational effects of these school-based performance award programs were not as 
strong as might be expected reinforces the idea that a successful award program involves more 
than promising school staff a bonus for meeting performance goals. As our motivation model 
implies, making a group reward program work is a complex undertaking, and difficulties estab- 
lishing any link weaken the potential motivational impact even when the other links are made. 
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Because of this complexity, school-based performance award programs are not likely to be suc- 
cessful unless considerable attention is paid to their design and management. It is important to 
note that the level of success of group reward programs in the private sector also varies. A siz- 
able minority of group incentive programs implemented in the private sector are judged unsuc- 
cessful (Abosch, 1998) or are discontinued (Bullock and Tubbs, 1990; Graham-Moore and Ross, 
1990). The detailed prescriptions given for private sector group reward programs also illustrate 
the complexity of designing them (for example, Orens and Elliott, 1997; Graham-Moore and 
Ross, 1990; Wilson and Phalen, 1996). This literature has typically outlined a set of conditions 
under which group rewards are more likely to work and has tended to be cautious in recommend- 
ing adoption where they are not present. Based on our research, and on research on private sector 
group reward programs, we recommend that designers of school-based performance awards at- 
tend to the following issues in order to improve motivational impact. 

1 . Teacher Expectancy. Our results show that schools with higher average teacher expectancy 
improve student achievement more, but also that many teachers perceived that the probability 
that their efforts would lead to their school’s meeting its improvement goals was not very 
high. Award program designers should therefore be concerned with ways to build expec- 
tancy. The motivation model highlights the importance of teachers’ perceptions of their com- 
petencies and the presence of enabling conditions. Though our measures of teacher compe- 
tence did not predict expectancy, both the private sector research literature (for example, 
Hatcher et al., 1991; Welboume and Gomez-Mejia, 1995) and the prescriptive literature (for 
example, O’Neill and Lough, 1994) on group reward programs have emphasized the impor- 
tance of showing workers how their efforts can influence the performance measures. Re- 
formers such as Cohen (1996) and the National Commission on Teaching and America’s Fu- 
ture (1996) have argued that many teachers lack the skills to teach to high standards. Profes- 
sional development focused on understanding the performance measures and the behaviors 
needed to influence them, as well as on providing the knowledge and skills needed to carry 
out the behaviors, may therefore help teachers believe that their efforts can lead to improved 
school performance. Pay incentives for acquiring the needed knowledge and skills may be a 
promising way to encourage teachers to develop the skills they need to believe they can suc- 
ceed, and therefore might complement school-based performance award programs. The 
other influence on expectancy in the model is the presence of enabling conditions such as 
principal leadership, professional community, and assessment feedback. Our research sug- 
gests that these are likely places to begin positively influencing teachers’ expectancy percep- 
tions. 

2. The Connection Between Goal Achievement and Receiving the Bonus. Our results show 
that many teachers were not very certain that if their school met its performance goals, the 
bonus would be paid out. Ways of strengthening teachers’ perceptions of the link between 
achievement and the bonus include institutionalizing a consistent source of funding and a 
commitment to paying the bonus when it is earned. For example, when North Carolina’s 
school-based performance award program was implemented in 1997, about $80 million was 
budgeted for the awards. Because more schools than expected met their improvement targets, 
full funding required $120 million. Rather than scaling back the level of the awards, the leg- 
islature increased the budget to pay all teachers in improving schools the awards that had 
been promised (Johnson et al. 1998). 
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3. The Size of the Bonus. Our interviews, particularly with Charlotte-Mecklenburg teachers, 
suggested that many teachers did not find the amount of the bonus large enough to be moti- 
vating. In order to reach a meaningful threshold for most teachers, bonuses in the $1,500- 
3,000 range may be needed. Larger amounts may help to offset some of the undesirable out- 
comes such as increased stress and longer hours. However, other reports (Hall and Caffarella, 
1997; Kelley, 1996; Kellor and Odden, 1998) suggest that teachers in some situations do see 
a $1,000 bonus as significant. The difference may be that in these cases, hours of work and 
stress levels may have been lower or perceptions of fairness higher. 

4. Active Support from Principals. Private sector literature has recognized the importance of 
the active commitment of site managers to the success of group reward programs (Bennett, 
1998; Graham-Moore and Ross, 1990; Ross, Hatcher, and Collins, 1992). Managers in suc- 
cessful private sector programs make the program central to their management efforts. They 
generate continuous interest and enthusiasm for the program. In both the Charlotte- 
Mecklenburg and Kentucky programs, this important role fell to principals, who may have 
had little or no guidance from the district or state as to how to carry it out. Based on our in- 
terviews, there appeared to be a high level of variability in the extent to which Charlotte- 
Mecklenburg and Kentucky principals promoted the program. While some principals were 
very proactive, others seemed to be genuinely at a loss as to how to rally their staff around 
the goals. 

5. Fairness Perceptions. The private sector research is clear about the importance of percep- 
tions of fairness on the motivational impact of reward programs (Cooper, Dyck, and Froh- 
lich, 1992; Gross and Barcher, 1993; Hatcher, Ross, and Collins, 1991; Lawler, 1988; Wel- 
boume and Gomez-Mejia, 1995). We also found that perceptions of fairness were important 
to teachers and influenced their motivational reactions. Perceived fairness was related to 
teacher expectancy, reported level of motivation by the bonus, desire to see the bonus pro- 
gram continue, and withdrawal intentions. To influence fairness perceptions, it may be nec- 
essary to expend considerable effort explaining the rationale behind the performance meas- 
ures, the level of the improvement goals, and the mechanics of the program. Reliability and 
face validity of student achievement assessments and other measures of school performance 
also are likely to be crucial. 

6. Teacher Participation in Award Program Design. Involving affected staff in the design of 
award programs might be a promising way to improve fairness perceptions, to ensure that 
improvement goals are not set too high as to seem unattainable, and to promote understand- 
ing and buy-in to the program. Again, private sector research (Abosch, 1998; Bullock and 
Tubbs, 1990; White, 1979) and prescription (O’Neill and Lough, 1994; Wilson and Phalen, 
1996) emphasizes the importance of participation in design and implementation. Kellor and 
Odden (1999) described a participative process used to design a school-based performance 
award program for the Cincinnati school district. 

Strengths and Limitations of Study Methods 

The results of our research and recommendations about program design should be interpreted 

within the context of the strengths and limitations of our methodology. Among the strengths is 
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the use of a teacher motivation model based on accepted motivational theory to serve as a guid- 
ing framework to study design and measurement, the use of a predictive design to assess the rela- 
tionship between teacher motivation and student achievement, the use of large samples from 
multiple research sites, and the use of qualitative and quantitative data collection and analysis. 
Analysis was also conducted at both the individual and school level to fully explore individual 
motivational reactions and the relation of motivation to performance. Because of these character- 
istics, we are confident that this research makes a substantive contribution to understanding how 
teachers function within, and respond to, a school-based performance award program in terms of 
their motivation to improve student achievement. 

Certain limitations of our study should be kept in mind. A first limitation pertains to measure- 
ment. Our measures for most of the independent variables were ad hoc in nature, a common 
characteristic of expectancy and goal-setting theory studies. As a result, very little is known 
about their construct validity. Additionally, operationalizing expectancy theory requires the use 
of multiplicative combinations of variables, which in turn exacerbates the unreliability of meas- 
urement. We suspect that such unreliability is the major cause of the weak relationship between 
the expected value of outcome measures and other variables. This is a particular problem in the 
analyses that used school-level averages of the expected value of outcomes to predict schools’ 
subsequent success in meeting their student achievement goals. That schools in which teachers 
valued the rewards associated with the award program do not appear to have higher levels of per- 
formance may be due to poor measurement rather than the lack of a substantive relationship. 

Our measures of school performance may also be subject to measurement error. We relied upon 
existing achievement measures used by the Charlotte-Mecklenburg and Kentucky programs. We 
had no reliability or validity data for the Charlotte-Mecklenburg measures. The information 
available on Kentucky’s KIRIS gain scores (Kentucky Department of Education, 1997) suggests 
that while they were fairly reliable, the scores were subject to a non-negligible degree of meas- 
urement error. This may have attenuated the predictive relationships between teacher motivation 
and subsequent school success. (See Milanowski, 1999 for a discussion of how the reported level 
of reliability affects the categorization of schools as eligible for receiving performance awards.) 

A second major limitation is our lack of control or comparison groups. Ideally, we would have 
studied the school-based performance award programs alongside identical programs without 
such key features as the bonus. By studying only sites that had award programs, we had to rely 
on naturally occurring variation within these programs (such as differences in teacher expec- 
tancy, perceived probability of receiving a bonus after meeting goals, and bonus desirability) to 
infer how effectively the programs motivated teachers and to estimate the impact of teacher mo- 
tivation on subsequent school success in meeting improvement goals. The availability of com- 
parison or control groups would have allowed us to better measure the true magnitude of the im- 
pact of award programs on teacher motivation and the impact of teacher motivation on school 
success. A definitive assessment of motivational value of an outcome such as the pay bonus may 
require an experimental design comparing the performance of two groups of schools operating 
under an identical accountability program, but with and without the pay bonus. 

A third limitation of our study pertains to the generalizability of our findings. The data were 
gathered midstream during the operation of the award programs, and it is uncertain whether our 
data and the relationships we found in them would have been the same at either earlier or later 
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points in time. For example, the Charlotte-Mecklenburg data were gathered as the Benchmark 
Goal Program was beginning to be integrated with a state-wide accountability plan. Now that the 
integration has been completed, we are uncertain as to how well our results generalize to this 
new circumstance. In Kentucky, we gathered our data at the time of substantial controversy over 
the state’s KIRIS testing process; had we gathered our data a year earlier, prior to the contro- 
versy, teachers’ responses might have been very different. Another constraint on generalizability 
is that we studied just three particular school-based performance award programs. Though we 
sampled carefully within our jurisdictions, thus enhancing generalizability to our jurisdiction 
populations, we have no way of knowing how well our results might generalize to award pro- 
grams in other jurisdictions. 



CPRE Research Report Series, RR-44 



44 



School-Based Performance Award Programs, Teacher Motivation, and School Performance 



REFERENCES 



Abelmaim, C. H., and Kenyon, S. B. (1996, April). Distractions from teaching and learning: 
Lessons from Kentucky ’s use of rewards. Paper presented at the annual meeting of the 
American Educational Research Association, New York, NY. 

Abosch, K. (1998, July/August). Variable pay: Do we have the basics in place? Compensation 
and Benefits Review, 12-22. 

Bennett, L. (1992, July/ August). Research from the trenchesDmaking group incentives work. 
Compensation and Benefits Review, 37-39. 

Blackford, L. B. (1997). Plan would shift KERA rewards. Lexington Herald-Leader [online]. 
Available : http ://www.kentuckyconnect/news/ 103197. 

Bullock, R. J., and Tubbs, M. E. (1991). A case meta-analysis of gainsharing plans or organiza- 
tional development interventions. Journal of Applied Behavioral Science, 26(3), 383- 
404. 

Cenziper, D. (1997, February 6). Paying to keep teachers. Charlotte-Mecklenburg Observer, 
lA. 

Cody, W. S. (1998, April 3). Memorandum to all Kentucky teachers. Available: 
http://www.kde.state.ky.us/coe/ocpg/dpi. 

Cohen, D. K. (1996). Rewarding teachers for student performance. In S. H. Fuhrman and J. A. 
O’Day (Eds.), Rewards and reform: Creating educational incentives that work. San 
Francisco, CA: Jossey-Bass. 

Cooper, C. L., Dyck, B., and Frohlich, N. (1992). Improving the effectiveness of gainsharing: 
The role of fairness and participation. Administrative Science Quarterly, 37, 471-490. 

Fuhrman, S. H. (1999). The new accountability. (CPRE Policy Brief RB-27). Philadelphia, PA: 
Consortium for Policy Research in Education, Graduate School of Education, University 
of Pennsylvania. ^ 

Fullan, M. G. (1991). The new meaning of educational change (X'^ Q^-)- New York, NY: 
Teachers College Press. 

Goertz, M. E., Massed, D. and Chun, T. (1997). Education reform policy: From Congress to the 
classroom. Maryland case study, 1996-1997 . Philadelphia, PA: Consortium for Policy 
Research in Education, Graduate School of Education, University of Pennsylvania. 

Gong, B. (personal communication, August 1, 1997). 



CPRE Research Report Series, RR-44 



45 




School-Based Performance Award Programs, Teacher Motivation, and School Performance 



Graham-Moore, B., and Ross, T. L. (1990). Gainsharing: Plans for improving performance. 
Washington, D.C.; Bureau of National Affairs, Inc. 

Gross, S. E., and Barcher, J. P. (1993, January/February). The new variable pay programs: How 
some succeed, why some don’t. Compensation and Benefits Review, 51-56. 

Hall, G. E., and Caffarella, E. P. (1997). Third year implementation assessment of the Douglas 
County, Colorado school district performance pay plan for teachers. No city: Authors. 

Hatcher, L., Ross, T. L., and Collins, D. (1991). Attributions for participation and nonparticipa- 
tion in gainsharing-plan involvement systems. Group and Organizational Studies, 16, 25- 
43. 

Hatry, H. P., Greiner, J. M., and Ashford, B. G. (1994). Issues and case studies in teacher in- 
centive plans (2"^ ed). Washington, D.C.: Urban Institute Press. 

Heneman, H. G., III. (1998). Assessment of the motivational reactions of teachers to a school- 
based performance award program. Journal of Personnel Evaluation in Education, 72(1), 
43-59. 

Heneman , H. G. Ill, and Judge, T. A. (in press). Compensation attitudes: A review and recom- 
mendations for further research. In S. L. Rynes and B. Gerhart (Eds.), Compensation in 
organizations: Progress and prospects. San Francisco, CA: New Lexington Press. 

Heneman, H. G. Ill, and Milanowski, A. (1998, November). Employees ’ withdrawal responses 
to their individual base pay and group bonus pay systems. Paper presented at the South- 
ern Management Association annual conference. New Orleans, LA. Unpublished manu- 
script, University of Wisconsin-Madison. 

Heneman, H. G. Ill, and Milanowski, A. (1999). Teacher attitudes about teacher bo- 
nuses under school-based performance award programs. Journal for Personnel 
Evaluation in Education, 72(4), 327-342. 

Heneman, H. G., Ill, and Schwab, D. P. (1985). Pay satisfaction: Its multidimensional nature 
and measurement. International Journal of Psychology, 20, 129-141. 

Heneman, R. L., Ledford, Jr., G. E., and Gresham, M. T. (in press). The effects of changes in 
the nature of work on compensation. In S. L. Rynes and B. Gerhart (Eds.), Compensa- 
tion in organizations: Progress and prospects. San Francisco, CA: New Lexington Press. 

Jacobson, S.L. (1987). Merit pay and teaching as a career. In K. Alexander and D.H. Monk 
(Eds.), Attracting and compensating America’s teachers (pp. 161-177). Eighth annual 
yearbook of the American Educational Finance Association, Cambridge, MA: Ballinger. 

Johnson, S.M. (1986). Incentives for teachers: What motivates, what matters? Educational Ad- 
ministration Quarterly, 22(3), 54-79. 



CPRE Research Report Series, RR-44 



46 



School-Based Performance Award Programs, Teacher Motivation, and School Performance 



Johnson, H., Leak, E., Williamson, G., Kellor, E. M., Odden, A. R., and Hanna, J. (1998) A case 
study of the state of North Carolina 's school-based performance award program. Unpub- 
lished manuscript. University of Wisconsin-Madison. 

Jones, A. P., Johnson, L. A., Butler, M. C., and Main, D. S. (1983). Apples and oranges: An 
empirical comparison of commonly used indices of interrater agreement. Academy of 
Management Journal, 2(5(3), 807-819. 

Kelley, C. (1996). Implementing teacher compensation reform in public schools: Lessons from 
the field. The Journal of School Business Management, 5(1), 37-54. 

Kelley, C. (1997). Teacher compensation and organization. Educational Evaluation and Policy 
Analysis, 19{\), 15-28. 

Kelley, C. (1998a, April). Schools that succeed: The role of enabling conditions in facilitating 
student achievement in school-based performance award programs. Paper presented at 
the American Educational Research Association annual conference, San Diego, CA. Un- 
published manuscript. University of Wisconsin-Madison. 

Kelley, C. (1998b). The Kentucky school-based performance award program: School- 
level effects. Educational Policy, 72(3), 305-324. 

Kelley, C., Conley, S., and Kimball, S. (1999). Incentives, professional norms, and local reali- 
ties: Evidence from Kentucky and Maryland. Paper prepared for the American Educa- 
tion Finance Association annual conference, Seattle, WA. Unpublished manuscript. Uni- 
versity of Wisconsin-Madison. 

Kelley, C., and Protsik, J. (1997). Risk and reward: Perspectives on the implementation 
of Kentucky’s school-based performance award program. Educational Admini- 
stration Quarterly, 33{4), 474-505. 

Kellor, E. M., and Odden , A. R. (1999). Cincinnati: A case study of the design of a school- 

based performance award program. Paper prepared for the American Education Finance 
Association annual conference, Seattle, WA. Unpublished manuscript. University of 
Wisconsin-Madison. 

Kellor, E. M., and Odden, A. R. (1998). Vaughn learning center: A case study. Available at the 
Consortium for Policy Research in Education, University of Wisconsin-Madison. 

Kentucky Department of Education. (1995, February). Kentucky teacher. Frankfort, KY: Au- 
thor. 

Kentucky Department of Education. (1996, May). Kentucky teacher. Frankfort, KY: Author. 

Kentucky Department of Education. (1997). KIRIS accountability cycle 2 technical manual 
(Chapter 1 1). Frankfort, KY: Author. 



CPRE Research Report Series, RR-44 



47 



School-Based Performance Award Programs, Teacher Motivation, and School Performance 



Lawler, E. E. Ill (1988). Gainsharing theory and research: Findings and future directions. Re- 
search in Organizational Change and Development, 2, 323-344. 

Lawler, E. E. III. (1990). Strategic pay: Aligning organizational strategies and pay systems. 

San Francisco, CA: Jossey-Bass. 

Lawton, M. (1997, July 9). Kentucky fires firm that ran innovative testing program. Education 
Week. 

Lee, T. W., and Mitchell, T. R. (1994). An alternative approach: The unfolding model of volun- 
tary employee turnover. Academy of Management Review, 19, 51-89. 

Locke, E. A., and Latham, G. P. (1990). A theory of goal setting and task performance. Engle- 
wood Cliffs, NJ: Prentice Hall. 

Maryland Department of Education. (1997). Maryland school performance report, 1997: State 
and school systems. Annapolis, MD: Author. 

Maryland Department of Education. (1999). Maryland school ‘report card’ shows solid gains; 
schools continue making progress on state test. Maryland State Department of Education 
Press Release. Annapolis, MD: Author. 

McAdams, J. C., and Hawk, E. J. (1994). Organizational performance and rewards: 663 ex- 
periences in making the link. Scottsdale, AZ: American Compensation Association. 

Milanowski, A. (1999). Measurement error or meaningful change? The consistency of school 
achievement in two school-based performance award programs. Journal of Personnel 
Evaluation in Education, 72(4), 343-363. 

Mumane, R. J., and Cohen, D. K. (1986). Merit pay and the evaluation problem: Why most 
merit pay plans fail and a few survive. Harvard Education Review, 56, 1-17. 

National Commission on Teaching and America’s Future. (1996). What matters most: Teaching 
for America 's future. Report of the National Commission on Teaching and America 's Fu- 
ture. New Y ork, NY : Author. 

Newman, F. M., and Associates, (1996). Authentic achievement: Restructuring schools for intel- 
lectual quality. San Francisco, CA: Jossey-Bass. 

Odden, A. R. (Ed.). (1991). Educational policy implementation. Albany, NY: State University 
of New York Press. 

Odden, A. R., and Kelley, C. (1997). Paying teachers for what they know and do: New and 

smarter compensation strategies to improve schools. Thousand Oaks, CA: Corwin Press. 



CPRE Research Report Series, RR-44 



48 



School-Based Performance Award Programs, Teacher Motivation, and School Performance 



O’Neill, D., and Lough, D. A. (1994). Team incentives and TQM: Building organizational excel- 
lence at National Semiconductor. American Compensation Association Journal, 3, 54-67. 

Orens, R. M., and Elliott, V. J. (1997). Variable pay programs: Pay for results. In P. T. Chingos, 
(Ed.), Paying for performance: A guide to compensation management. New York, NY : 
Wiley. 

Reidy, E. (1995, September). The Kentucky Education Reform Act and school-based perform- 
ance awards. Paper presented at the CPRE Conference Series on Teacher Compensation, 
Washington, D.C. 

Rosenholtz, S.J. (1989). Teachers ’ workplace: The social organization of schools. White 
Plains, NY : Longman. 

Ross, T. L., Hatcher, L., and Collins, D. (1992). Why employees support (and oppose) gainshar- 
ing plans. Compensation and Benefits Management, 8, 17-27. 

Stecher, B. M., and Barron, S. (1999). Test-based accountability: The perverse consequences of 
milepost testing. Paper presented at the annual meeting of the American Educational Re- 
search Association, Montreal, Canada. 

Strauss, A., and Corbin, J. (1994). Grounded theory methodology: An overview. In N. K. Den- 
zin and Y.S. Lincoln (Eds.), Handbook of qualitative research (pp. 273-285). Thousand 
Oaks, CA: Sage Publications. 

Strecklow, S. A. (1997, September 2). Kentucky’s teachers get bonuses, but some are caught 
cheating. Wall Street Journal, A1 . 

Tschannen-Moran, M., Hoy, A. W., and Hoy, W. K. (1998). Teacher efficacy: Its meaning and 
measure. Review of Education Research, 68(2), 202-248. 

Van Eerde, W., and Thierry, H. (1996). Vroom’s expectancy models and work-related criteria: 

A meta-analysis. Journal of Applied Psychology, 81(5), 575-586. 

Vroom, V. H. (1964). Work and motivation. New York, NY: Wiley. 

Welboume. T. M. and Gomez-Mejia, L. R. (1995). Gainsharing: A critical review and a future 
research agenda. Journal of Management, 21, 559-609. 

White, J. K. (1979). The Scanlon plan: Causes and correlates of success. Academy of Manage- 
ment Journal, 22, 293-312. 

Wilson, T. B., and Phalen, C. P. (1996). Rewarding group performance: An approach to design- 
ing and implementing incentive pay programs. Scottsdale, AZ: American Compensation 
Association. 



CPRE Research Report Series, RR-44 



49 



School-Based Performance Award Programs, Teacher Motivation, and School Performance 



TECHNICAL APPENDIX 



Further Description of Data Collection Activities 

In each jurisdiction, data collection involved on-site interviews with principals and teachers in 
schools with varying degrees of success in meeting school performance goals, followed by mail 
surveys of teachers (in Kentucky and Charlotte-Mecklenburg) and principals (in all three juris- 
dictions) in a larger sample of schools. Table A1 summarizes the nature and timing of the data 
collection effort. 



Table A1. Summary of Data Collection Activities: Studies of School-Based Performance Awards 



SBPA Program 


Timeframe 


Method 


Sample 


Study Focus 


Kentucky 

Accountability 

Program 


1995 


Interviews (1/2 day 
at each school) 


Teachers, principals in 6 
reward elementary and 
middle schools; state 
dept, of ed., state stake- 
holders 


Motivation model; 
program effects 




1996 


Interviews (1/2 day 
at each school) 


Teachers, principals in 
16 elementary, middle 
and high schools with 
varying levels of success 
in the program 


Motivation model; 
program effects 




1997 


Teachers sur- 
veyed by mail 


Up to 25 teachers in 262 
elementary, middle, and 
high schools: 5654 sent 
questionnaires, 1750 
responded, for a 30.9% 
response rate 


Motivation model; explana- 
tion for differences in 
achievement outcomes 
across schools; program 
effects 




1997 


Principals sur- 
veyed by mail 


393 principals sent ques- 
tionnaires, 204 re- 
sponded, for a 51.9% 
response rate 


Explanation for differences 
in achievement across 
schools; program effects 


Charlotte- 
Mecklenburg 
Benchmark Goals 
Program 


1997 


Interviews (1/2 day 
at each school) 


Teachers and principals 
in 11 elem., middle and 
high schools with varying 
levels of success in the 
program; district level 
staff 


Motivation model; 
program effects 




1997 


Teachers sur- 
veyed by mail 


Up to 25 teachers in 120 
elementary, middle, and 
high schools; 2951 
teachers sent question- 
naires, 1150 responded, 
for a 39.0% response 
rate 


Motivation model; explana- 
tion for differences in 
achievement outcomes 
across schools; program 
effects 




1997 


Principals sur- 
veyed by mail 


120 principals sent ques- 
tionnaires, 74 responded 
fora 61.6% response 
rate 


Explanation for differences 
in achievement across 
schools; program effects 




56 
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SBPA Program 


Timeframe 


Method 


Sample 


Study Focus 


Maryland Reward 
for Success 


1997 


Interviews (1/2 day 
at each school); 
review of docu- 
ments 


Teachers and principals 
in 11 elem. and middle 
schools with varying lev- 
els of success in the pro- 
gram; state and district- 
level administrators 


Motivation model; program 
effects as compared to 
Charlotte-Mecklenburg and 
Kentucky 




1997 


Principals sur- 
veyed by mail 


430 principals sent ques- 
tionnaires, 236 re- 
sponded for a 54.9% 
response rate 


Explanation for differences 
in achievement across 
schools; program effects 



On-site Interviews 

An initial round of interviews was conducted in a purposive sample of six Kentucky reward 
schools in 1995, followed by a larger sample of 16 schools in Kentucky in 1996. Interviews in 
Charlotte-Mecklenburg were conducted in 12 schools in 1997 and in 1 1 schools in Maryland in 
1998. The schools in the 1996, 1997, and 1998 studies were chosen to include those with a vari- 
ety of rates of success with the two reward programs. The 1996 Kentucky sample was selected to 
include schools from each level (elementary, middle, and high schools) and all possible combi- 
nations across the first two accountability cycles of winning monetary awards (reward), winning 
neither awards nor receiving sanctions (nonreward), and receiving sanctions (sanction). Assign- 
ment of schools into three categories rather than the five categories used by the Kentucky De- 
partment of Education was based on our interest in understanding the effects of monetary awards 
and sanctions on teacher motivation and on our 1995 research in Kentucky, which suggested that 
teachers viewed the “real” reward as exceeding the goal and receiving the monetary award and 
the “real” sanction as occurring in the decline and crisis categories. This created six different pat- 
terns of award histories in Kentucky: reward/reward, nonreward/reward, sanction/reward, re- 
ward/nonreward, nonreward/nonreward, and reward/sanction. There were no schools in the sanc- 
tion/sanction category, so this category was not included in our sample. 

In Charlotte-Mecklenburg, the purposive sample included schools that received rewards in both 
1994-95 and 1995-96 (at the 100 percent or 75 percent level), schools that were bonus winners in 
only 1994-95, and schools that did not receive bonus awards in either time period. 

In Maryland, 1 1 elementary and middle schools from two districts were selected for study. High 
schools were not included because at the time of the study, they did not participate in the award 
program. The sample of schools included both elementary and middle schools in each district 
that had made improvements for two consecutive years (reward), those that made one year of 
improvement (certificate), and those that had declined in performance or failed to improve 
enough to be in an award category. 

In all three jurisdictions, schools were selected for diversity with regard to socioeconomic status, 
geographic location, and in Charlotte-Mecklenburg, racial composition and magnet/nonmagnet 
status. Researchers spent a half-day in each school interviewing the principal and a sample of 
teachers, particularly those who taught in the accountability subject/grades. Typical interviews 
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lasted 45 to 60 minutes. The total number of teachers interviewed was approximately 105 in 
Kentucky, 60 in Charlotte-Mecklenburg, and 60 in Maryland. In general, interviews with teach- 
ers were done in a group setting, with between 3 and 10 teachers participating. In all three sites, 
official program documents were examined. In Kentucky and Maryland, information was gath- 
ered from state-level personnel to better understand program context, design, and implementa- 
tion. In Maryland and Charlotte-Mecklenburg, interviews also were conducted with district- 
level personnel. (District-level personnel were not typically interviewed in Kentucky, since the 
Kentucky program was largely designed to bypass districts and operate directly at the school 
level.) 

Interview protocols were used to guide the interviews and provide consistency across research 
teams. Typically, two researchers worked together at each research site to provide greater flexi- 
bility in data collection and to help interpret findings. Over all, four researchers collected data at 
each research site. Immediately following each school visit, researchers recorded initial percep- 
tions on a summary sheet. Findings from the early data collection and analysis efforts helped to 
inform later data collection efforts, both within and across research sites. Interviews with teach- 
ers and principals were taped, and the tapes were transcribed. Transcripts were content-analyzed 
to identify the major themes of each interview. A coding scheme was developed by having four 
different researchers who were involved in the data collection code one set of transcripts and to- 
gether develop a coding scheme that was then used by 2 researchers to code the remaining data. 
Coding schemes were developed for Kentucky, Charlotte-Mecklenburg, and Maryland. The 
codes were allowed to emerge from the data, but coding schemes in Charlotte-Mecklenburg and 
Maryland were informed by the earlier work. Transcripts were analyzed for patterns across 
school levels, schools with different levels of success with the award program, and magnet ver- 
sus non-magnet schools. 

Survey Research 

Findings from the qualitative phase of the research, along with research literature on high- 
performance organizations, effective schools, and teacher/employee motivation were used to de- 
velop the survey instruments. Separate instruments were developed for teachers and principals. 
Where possible, scale items were adapted from pre-existing survey instruments. Common ques- 
tions were included across instruments, although the questions were tailored to be descriptive of 
the programs operating in each research site. 

The teacher questionnaire used in both Charlotte-Mecklenburg and Kentucky contained a series 
of questions designed to measure the constructs of the motivation model (goal commitment, clar- 
ity, and conflict, expectancy perceptions, probability and valence of outcomes associated with 
the program, teaching skills and attitudes, organizational supports for achievement of program 
goals). Also, perceptions of program fairness, satisfaction with pay, effect of the program on in- 
tent to leave the school, and demographic items. In addition, teachers were given an opportunity 
to provide comments at the end of the questionnaire. The principal questionnaire was a shorter 
document, focusing on organizational supports for achievement of program goals, but also in- 
cluding questions on goal commitment, clarity, conflict, expectancy perceptions, and attitudes 
toward the accountability program. 
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The questionnaires were administered to Charlotte-Mecklenburg teachers and principals in the 
spring of 1997. The teacher questionnaire was administered to 25 teachers selected randomly 
from 120 schools. (Ten small schools serving specialized populations were excluded.) If a school 
had 25 or fewer teachers, all of the teachers were selected. The initial sample included 2,95 1 
teachers. Using district-supplied home address mailing labels, the survey questionnaires were 
sent out in mid- April of 1997. Two follow-up mailings were made. All questionnaires were re^ 
turned in addressed, postage-paid envelopes to the Wisconsin Survey Research Laboratory at the 
University of Wisconsin-Madison. By the July 15 cutoff date, 1150 teachers had responded, for a 
39 percent response rate. The responders matched the Charlotte-Mecklenburg teacher population 
relatively well by gender, race, and teaching service, but elementary teachers were somewhat 
over-represented and high school teachers under-represented. The principal questionnaire was 
sent to each of the 120 Charlotte-Mecklenburg principals, who were requested to return them to 
the CPRE Wisconsin office. By the July 15 cut-off, and after two rounds of follow-ups by phone, 
76 principals had provided usable responses, for a response rate of 63 percent. Slightly more 
elementary schools were represented, at the expense of middle schools, and principals in reward 
schools were slightly more likely to respond than principals in nonreward schools. 

In Kentucky, teachers and principals were sent questionnaires in the fall of 1997. Respondents 
were selected by means of a multi-stage sampling process based on a list of schools provided by 
the Kentucky Department of Education. First, schools were selected within strata based on five 
patterns of prior accountability status (e.g., reward both cycles, nonreward both cycles) and 
school level (high, middle or junior high, and elementary). There were a total of 15 strata. 

Within each strata, a random sample of 40 schools was chosen (or all the schools were chosen if 
there were 40 or fewer schools in the stratum). The total number of schools selected at this stage 
was 392. Because no single list of teacher names was available, districts or schools were con- 
tacted and asked to provide names of teachers. At this stage, districts or schools agreed to pro- 
vide teacher names for 262 schools. Within these schools, a random sample of 25 teachers was 
taken. If there were 25 or fewer teachers in a school, all teachers were included. The 5,654 
teachers in the sample were sent questionnaires by mail in late October of 1997. Two follow-up 
mailings were sent to teachers who did not respond to the initial questionnaire. All question- 
naires were returned to the Wisconsin Survey Research Laboratory at the University of Wiscon- 
sin-Madison. The total number of useable responses received was 1,750, for a response rate, rela- 
tive to the number of questionnaires sent, of 30.9 percent. At least one response was received 
from 261 of the 262 schools. (The estimated number of teachers who would have been sent ques- 
tionnaires had all schools in the first stage sample provided names was 8,136.) At the school 
level, responses were obtained from 67 percent of the schools in the original sample. The per- 
centage of the strata from which teacher responses were obtained ranged from 53 to 100%. 

Only a very limited amount of information was available upon which to base a comparison of 
respondent demographic characteristics to characteristics of the overall Kentucky teacher popula- 
tion. The distribution of respondents in high schools versus elementary and middle schools was 
close to the distribution reported in the Common Core of Data (National Center for Education 
Statistics, 1996) for the 1994 school year. According to data obtained from the Kentucky De- 
partment of Education, the average annual teacher salary was $32,807 in 1997. The median pay 
level in our sample was in the $30,000 to $34,999 category. Because the sample was intended to 
over-represent some types of schools (e.g. Decline/Reward) relative to others (Nonreward both 
cycles), it likely contains more teachers from schools that achieved reward status in Cycle 2. 
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The principals of each of the Kentucky schools selected in the first stage of the sampling process 
described above also were sent questionnaires in the fall of 1997. Questionnaires were mailed 
from the CPRE Wisconsin office and were returned there. After two sets of reminder mailings, 
of the 392 principals contacted, 204 returned useable questionnaires, for a response rate of 52 
percent. Coverage within-strata ranged from 31 to 71 percent. 

In Maryland, principals were sent questionnaires in the spring of 1998. A sample size of 500 was 
allocated equally between middle and elementary schools. A simple random sample of 250 
schools was taken within each strata, based on mailing lists provided by the Maryland Depart- 
ment of Education. After the initial mailing, the district of Prince Georges County asked that 
their principals not be contacted. Responses received from this district were deleted from subse- 
quent analysis and destroyed. Eight of the schools on the mailing list did not participate in the 
program, so they were removed from the sample. Usable responses were received from 252 prin- 
cipals, for a response rate of 51 percent (252 divided by 500 minus 8). 

Additional data on schools were collected from documents and data files provided by the juris- 
dictions. The Charlotte-Mecklenburg district supplied information on the following variables: 
enrollment, percent of students receiving free and reduced priced lunch, and Benchmark Goals 
point totals and reward status for each of the years of program operation. Data files allowing 
identification of which teachers transferred schools or left the district also were provided. The 
Kentucky Department of Education supplied data on school enrollment and percent of students 
receiving free and reduced priced lunch. The KIRIS scores and prior accountability status of 
schools were obtained from the Kentucky Department of Education. 

In most of the analyses reported, it was necessary to take into account the fact that simple ran- 
dom sampling was not used. At the individual teacher level, the probability of selection was not 
equal for individual respondents. In Charlotte, selection probabilities varied by school size. For 
Kentucky, selection probabilities for schools (the primary sampling unit) differed by strata, and 
selection probabilities for teachers within schools differed by school size. To reflect these differ- 
ences, most of the statistical analyses were conducted using case weights proportional to the in- 
verse of the selection probability. Additionally, since the sampling technique selects first 
schools, then teachers within schools, teachers in the sample were not independent picks from 
the populations. Standard errors and significance tests for analyses at the individual teacher level 
need to take this into account, and so most individual-level analyses were done using regression 
routines that adjusted standard errors to reflect this clustering. 
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Table A2. Items Combined to Produce Goal Commitment, Goal Clarity, Goal Focus, and Goal Conflict 

Scales and the Internal Consistency Reliability of the Scales 

Coefficient 

Scale and Items Alpha 



Goal Commitment 

Charlotte-Mecklenburg .73 

It’s hard to take the Benchmark Goals seriously, (scale reversed) 

I am strongly committed to pursuing my school’s Benchmark Goals. 

Quite frankly, I don’t care if my school achieves its Benchmark Goals or not. 

(scale reversed) 

Kentucky -74 

It’s hard to take the accountability goals seriously, (scale reversed) 

I am strongly committed to pursuing my school’s accountability goals. 

Quite frankly, I don’t care if my school achieves its accountability goals or not. (scale reversed) 

Goal Clarity 

Charlotte-Mecklenburg .90 

1 have a clear understanding of what my school’s Benchmark Goals are. 

I know how much each aspect of student achievement needs to improve in order to reach the 
Benchmark Goals for this year. 

I have a good understanding of how student achievement related to the Benchmark Goals is 
measured in the subject(s) I teach. 

I could explain my school’s Benchmark Goals to others if asked. 

Kentucky -85 

1 have a clear understanding of what my school’s accountability goals are. 

1 know how much each aspect of student achievement needs to improve in order to reach the ac- 
countability goals for this cycle. 

I have a good understanding of how student achievement is measured by KIRIS in the subject(s) I 
teach. 

1 could explain my school’s accountability goals to others if asked. 

Goal Focus -69 

Kentucky 

The accountability goals provide a focus for my teaching efforts. 

The accountability goals tell us what is most important for the school to accomplish. 

Goal Conflict 

Charlotte-Mecklenburg .75 

This year’s Benchmark Goals conflict with other important goals of our school. 

Working toward our school’s Benchmark Goals prevents me from working toward other important 
educational goals. 

Kentucky .80 

The accountability goals conflict with other Important goals of our school. 

Working toward our school’s accountability goals prevents me from working toward other important 
educational goals. 
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Table A3. Relationship of Expected Value of Positive Outcomes of School-Based Performance Award 



Programs to Goal Clarity, Commitment, and Choice of Goal Level 



1. Charlotte-Mecklenburg 


Goal Clarity 


Goal Commitment 


Years Experience 


.195*** 




.145** 


Teach Subject with Benchmark Goal 


.091** 




.096** 


Assessment? 








High School 


-.120 




-.117* 


Middle School 


-.141** 




-.094** 


Percent Free/Reduced-Price Lunch 


.007 




-.001 


Natural Log of Enrollment 


.041 




-.045 


Pay Level 


.002 




.028 


Pay Level Satisfaction 


-.012 




-.015 


Procedural Fairness 


.182** 




.249*** 


Individual Expectancy 


.160*** 




.150*** 


Expected Value of Bonus 


.092*** 




.208** 


N 


965 




962 


R-squared 


.169 




.256 




II. Kentucky 


Goal 


Goal 


Goal 




Clarity Commitment 


Focus 


Years Experience 


.050 


.031 


.025 


Teach Subject with KIRIS Assessment? 


.103*** 


.032 


.064* 


Teach Grade with KIRIS Assessment? 


-.017 


-.001 


-.003 


High School 


-.040 


-.005 


-.045 


Middle School 


-.010 


.048 


.054* 


Percent Free/Reduced-Price Lunch 


-.009 


.063 


-.053 


Natural Log of Enrollment 


-.052 


-.036 


- 123*** 


Prior Cycle KIRIS Score 


-.017 


.004 


-.062** 


Pay Level 


.085* 


-.039 


-.005 


Pay Level Satisfaction 


.069** 


.056** 


.026 


Procedural Fairness 


.170*** 


.229*** 


.294*** 


Individual Expectancy 


.152*** 


.104*** 


.093*** 


Expected Value of Bonus 


.045 


.115*** 


.082*** 


Sum of Expected Value of Goal Attain- 








ment Outcomes 








n 


1542 


1538 


1543 


R-squared 


.110 


.130 


.157 



* significant at the . 10 level 
** significant at the .05 level 
♦♦♦ significant at the .01 level 



Comment on Table A3 

The control variables of teaching experience (whether teaching a subject or grade in which an 
accountability assessment is given), school level, school percent on free/reduced-price lunch, 
school size, perceptions of program fairness were included because they represent factors that are 
likely to affect commitment or clarity. Pay level, pay level satisfaction, and individual expec- 
tancy are included because they are likely to affect the motivational value of the bonus. For the 
Kentucky analysis, an additional control variableDthe value of the prior cycle’s accountability 
indexDwas added since it might be harder to improve at higher absolute levels of achievement, 
and this could in turn have a negative effect on goal commitment and focus. 
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Table A4. Standard Deviations of Probability Ratings (Instrumentalities) of Outcomes Associated with 

School-Based Performance Award Programs 

Outcome Probability if Probability if You Try Probability if School 

School Meets Goals to Meet Goals Does Not Meet 

Goals 





Charlotte 


KY 


Charlotte 


KY 


Charlotte 


KY 


Receiving a bonus 


30.9 


35.3 










Personal satisfaction of meeting goals 
Personal satisfaction from improved student 


28.1 


28.0 










performance 


25.8 


26.4 










Public recognition for meeting goals 


31.1 


30.9 










Receiving funds for school improvements 


30.3 


30.8 










Working cooperatively with other teachers 
on curriculum and instruction 






24.1 


22.9 






Feel you are working toward clear school- 
wide goals 






27.7 


27.2 






Having additional opportunities for profes- 
sional development 






27.1 


23.8 






Having your students learn new skills 






29.1 


26.5 






Feel you are participation in education re- 
form program 






30.0 


30.1 






More pressure and job stress 






26.2 


20.3 






Putting in more hours 






29.3 


22.1 






Less freedom to teach things unrelated to 
goals 






27.0 


24.6 






Public criticism or embarrassment due to 
not achieving goals 










31.8 


29.2 


Intervention 










32.4 


28.0 


Loss of professional pride due to not 
achieving goals 










33.0 


31.5 


Risk to job security 










29.5 


31.6 



* $1,000 in Charlotte- Mecklenburg, $2,000 in Kentucky 
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Table A5. Items Used in Scales for Analysis of Teacher Expectancy 

Dependent Variables: 

Individual Teacher Expectancy 

Charlotte-Mecklenburg: If you were to make the highest level of effort you would be comfortable with (that you 
could sustain for the whole year), what is the probability or likelihood of your school meeting its Benchmark Goals? 
(0%, 10%, 20%,. . . 100%) 

Kentucky: What is the chance, probability, or likelihood that if you personally were to make the highest level of 
effort you could, your school would exceed its accountability goals by enough to be a “Reward” school? (0%, 10%, 
20%,. . . 100%) 



Teacher Group Expectancy: 

Charlotte-Mecklenburg: If you and your colleagues were to work together as hard as you could, what is the probabil- 
ity or likelihood of your school meeting its Benchmark Goals? (0%, 10%, 20%, . . . 100%) 

Kentucky: What is the chance, probability, or likelihood that if you and your colleagues were to work together as 
hard as you could, your school would exceed its accountability g oals by enough to be a “Reward” school? (0%, 
10%, 20%, . . . 100%) 

Independent Variables: 

School Characteristics: 

School Size: Student Headcount Enrollment 
Free Lunch: Percent Free and Reduced-Price Lunch 

Reward History: for Charlotte-Mecklenburg, 1995-96 performance on the Benchmark Goals (Reward=l , Nonre- 
ward=0); for Kentucky: Cycle 2 performance on the Kentucky School-Based Accountability Program (Reward=l, 
Nonreward=0) 

Enabling Conditions: Scales created from responses to the following five-point likert scale questions: 

Feedback 

Charlotte-Mecklenburg: Concerning the feedback you receive about the Benchmark Goals Program, to what extent: 

- Do you receive the results of the tests or other assessments that are used to measure whether your school achieved 

its Benchmark Goals in the subjects you teach? 

- Do you know how to interpret the results of the assessments related to the Benchmark Goals? 

- Are the results of these assessments useful to you in figuring out how to help achieve your school’s Benchmark 
Goals? 

Kentucky: Concerning the feedback you receive about the accountability goals, to what extent: 

- Do you receive KIRIS results relevant to the subjects you teach? 

- Do you know how to interpret the results of the KIRIS assessments in relation to the accountability goals? 

- Are the results of the KIRIS assessments useful to you in figuring out how to help achieve your school’s account- 

ability goals? 

Alignment 

Charlotte-Mecklenburg: At your school, to what extent: 

- are teachers evaluated based on their efforts to achieve the school’s Benchmark Goals? 

- are new curriculum development efforts linked to the Benchmark Goals assessment? 

- are teachers provided time to plan together about how to achieve the Benchmark Goals? 
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- has your school mobilized its resources to meet its Benchmark Goals? 

- do professional development activities in your school focus on helping you acquire knowledge and skills related to 
meeting the Benchmark Goals? 

Kentucky: At your school, to what extent: 

- Are teachers evaluated based on their efforts to achieve the school’s accountability goals? 

- Are new curriculum development efforts linked to KIRIS? 

- Are teachers provided time to plan together about how to achieve the accountability goals? 

- Has your school mobilized its resources to meet its accountability goals? 

- Do professional development activities in your school focus on helping you acquire knowledge and skills related to 
meeting the accountability goals? 

Community (a): At your school, to what extent do teachers in your school work together to improve teaching 
knowledge and skills? 

Community (b) (for Kentucky only): 

At your school, to what extent do teachers in your school work together to improve teaching knowledge and skills? 
Do teachers seek the advice of other teachers when they have a teaching problem? 

Do teachers discuss ideas about how to improve teaching and student learning among themselves? 

Do teachers share a vision of student learning and what it takes to accomplish it? 

Do teachers hold one another accountable to work hard to increase student learning? 

Teacher Participation: At your school, to what extent: 

- Do teachers make important decisions about curriculum and instruction? 

- Do teachers make important decisions about the allocation of the school’s budget? 

- Do teachers make important decisions about the hiring of new teachers? 

- Are teachers provided with the information they need to participate meaningfully in school-level management de- 

cisions? 

Principal Support 

Charlotte-Mecklenburg: To what extent is your school’s principal: 

- committed to achieving your school’s Benchmark Goals? 

- supportive of your efforts to achieve the school’s Benchmark Goals? 

- willing to work with teachers to help them understand how to achieve the Benchmark Goals? 

Kentucky: To what extent is your school’s principal: 

- committed to achieving your school’s accountability goals? 

- supportive of your efforts to achieve the school’s accountability goals? 

- willing to work with teachers to help them understand how to achieve the accountability goals? 

Goal Conflict (extent of agreement or disagreement with the following statements) 

Charlotte-Mecklenburg: 

- This year’s Benchmark Goals conflict with other important goals of our school. 

- Working toward our school’s Benchmark Goals prevents me from working toward other important educational 

goals. 

Kentucky: 

- The accountability goals conflict with other important goals of our school. 

- Working toward our school’s accountability goals prevents me from working toward other important educational 

goals. 

Teacher Attitudes: Scale created from responses to the following five-point likert scale questions: 
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Fairness (extent of agreement or disagreement with the following statements) 

Charlotte-Mecklenburg: 

- The way our school’s Benchmark Goals were established was fair. 

- As a way to measure school performance, the Benchmark Goals are fair. 

- Overall, the way the Benchmark Goals program is run is fair. 

Kentucky: 

- The way our school’s accountability goals were established was fair. 

- As a way to measure school performance, the accountability goals are fair. 

- Overall, the way the accountability program is run is fair. 

Goal Clarity (extent of agreement or disagreement with the following statements): 

Charlotte-Mecklenburg: 

- 1 have a clear understanding of what my school’s Benchmark Goals are. 

- 1 know how much each aspect of student achievement needs to improve in order to reach the Benchmark Goals for 
this year. 

- 1 have a good understanding of how student achievement related to the Benchmark Goals is measured in the sub- 
ject(s) I teach. 

- 1 could explain my school’s Benchmark Goals to others if asked. 

Kentucky : 

- 1 have a clear understanding of what my school’s accountability goals are. 

- 1 know how much each aspect of student achievement needs to improve in order to reach the accountability goals 
for this cycle. 

- 1 have a good understanding of how student achievement is measured by KIRIS in the subject(s) I teach. 

- 1 could explain my school’s accountability goals to others if asked. 

Teacher Characteristics 

Experience : Total number of years of teaching experience. 

Education : 1= Highest degree obtained is MA/MS, MA/MS plus credits, or EdD or PhD, else = 0 
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Table A7. Items Combined to Produce Scales Used in the Analysis of the Motivational Reactions of a 
Subsample of Kentucky Teachers to Features of the School-Based Performance Award Program and 

Internal Consistency Reliabilities 



Scale and Items 


Coefficient 

Alpha 


Fair Bonus 

It is fair for teachers who Increase student achievement to receive a bonus. 
The size of the bonus 1 could receive is fair. 


.71 


Likely Bonus Value 

Rated desirability of winning a $2,000 bonus muliplied by likelihood of winning the bonus 


NA 


Unfair Winners 

Some schools that do not deserve it are winning bonuses. 
Some individuals that do not deserve it are winning bonuses 


.82 


Fair Chance 

Our school has as much chance of being a reward school as any other. 

Our school has less chance of achieving is accountability goals because of our student 
population. 


.73 


Fair Process 

As a way to measure school performance, the accountability goals are fair. 

The standards (accountability goals) for my school to get a reward are fair. 

The bonus part of the accountability system is administered fairly. 

The appeal process a school can use to challenge its accountability results is fair. 
The way our school’s accountability goals were established is fair. 


.81 


Teachers Divide 

Only teachers should decide how to divide up the bonus money. 

Only teachers should decide how to divide up the reward money between bonuses and school 
Improvements. 


.97 


Principal Communication 

Our principal works with us to achieve our accountability goals. 

There has been adequate communication to teachers about the bonus process and program. 


,55 


Pay Satisfaction 

[How satisfied are you with] Your take home pay 
[How satisfied are you with] Your current salary 
[How satisfied are you with] Your overall pay level 


.97 


Bonus Motivation 

1 would work just as hard to achieve our school’s accountability goals even without the 
possibility of getting a bonus, (reverse scored) 

If the bonus were discontinued, my support for the accountability program would decrease. 
Tripling the size of the bonus would greatly increase my motivation to meet our school’s 
accountability goals. 


.62 


Bonus Continuation 

The bonus part of the accountability program should be continued. 




Withdrawal Intentions 

[How likely are you to] Move to another teaching job in another school or district. 

[How likely are you to] Move to a teaching job in a school that has been more successful In 
meeting its goals. 

[How likely are you to] Leave teaching for a different type of work. 

[How likely are you to] Retire before you reach normal retirement age. 


.80 
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Table A10. Items Combined to Produce Fairness of School-Based Performance Award Programs Scales 

and the Internal Consistency Reliability of the Scales 





Charlotte 


Kentucky 


Scale and Items 


Coefficient 


Coefficient 




Alpha 


Alpha 


School has a fair chance to win award (2 item scale) 


.78 


.71 


Our school has as much chance of being a reward school as any other. 

Our school has less chance of achieving its accountability goals than others 
Because of our student population, (scale reversed) 






Program is procedurally fair 

As a way to measure school performance, the accountability goals are fair. 
The way our school’s accountability goals were established is fair. 

Overall, the way the accountability program is run is fair. 


.85 


.84 



Technical Issues in School-Level Analysis of School Performance 

As mentioned in the body of the paper, analyses of the effect of teacher motivational responses 
were done at the school level. This involved aggregating individual teacher responses by calcu- 
lating the mean for each school of the scale or variable of interest. Unfortunately, there were a 
number of schools in which only a few teachers responded. A decision was made to drop schools 
from any school-level analyses if there were not at least four respondents per school. This meant 
dropping two schools from the Charlotte-Mecklenburg analysis and 50 schools from the Ken- 
tucky analysis. Additional schools also were dropped for both sites because information about 
some other variable needed for a specific analysis was not available for that school. This meant 
that for Charlotte, the sample size for school-level analyses was 1 14, and for Kentucky, 204. For 
the Charlotte schools, the average number of respondents per school for these schools was 9.6, 
while for the Kentucky schools, it was 7.8. 

Whenever individual-level data are aggregated to form a score purporting to represent a higher- 
level entity such as a school, the question of the degree of agreement in the responses of the in- 
dividuals arises. Here we are concerned with the within-school averages of teachers’ ratings of 
their motivational responses, which we are using to represent the average level of these con- 
structs within each school. Within-school variability in individual teachers’ responses leads to 
sampling error in each school-level average. One way to assess within-school variability is to 
calculate the intraclass correlation coefficient (ICC) within-school and use this coefficient to es- 
timate the reliability of the school average using the Spearman-Brown formula (Nunnally, 1978). 
Table 1 1 shows the ICCs and reliability estimates for the motivational variables. 
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Table A11. Intraclass Correlations, Reliabilities, and Average Within-School Standard Errors for School- 

Level Expectancy Theory Variables 



Variable 


Charlotte 

ICC 


Charlotte 
Reliability 
of Within- 
School 
Avg, 


Charlotte 

Average 

Within- 

School 

Stand. 

Err. 


Kentucky 

ICC 


Kentucky 
Reliability 
of Within- 
School 
Avg. 


Kentucky 

Average 

Within- 

School 

Stand. 

Err. 


Individual Expectancy - Reward 


.093 


.479 


9.62 


.115 


.496 


9.08 


Individual Expectancy-Avoid Sanc- 


- 


- 




.051 


.289 


9.60 


tions 

Expected Value - Bonus 


.072 


.421 


.24 


.130 


.533 


.27 


Expected Value - Goal Attainment 


.087 


.468 


.71 


.089 


.423 


.77 


Outcomes 

Expected Value - Learning Out- 


.075 


.429 


.79 


.090 


.426 


.86 


comes 

Expected Value - Sanctions 


.004 


.038 


.78 


.041 


.245 


.82 


Expected Value - Stress Outcomes 


.058 


.363 


.57 


.029 


.183 


.67 



The ICCs in the table can be interpreted as the percent of variation in teachers’ response levels 
that is accounted for by working in the same school. The reliability coefficient can be interpreted 
as the expected correlation between average scorers across different samples of the same size 
within the average school in the sample. While the ICCs show that there is significant between- 
school variation in the motivational reactions (all of the underlying ANOVA’s show a between- 
school effect significant at the .05 level or beyond, except for the expected value of sanctions for 
Charlotte-Mecklenburg) the proportion of variation attributable to schools is relatively low. 

While we did not expect to see a large school effect for the expected value variables (in part, be- 
cause there is no a priori reason to think that teachers differ greatly in what they value across 
schools), the small school effect on expectancy was not expected, since the motivational model 
and prior results at the individual level suggest that what happens at the school level (e.g., prin- 
cipal leadership and feedback) helps determine expectancy perceptions. 

However, it should be recognized that ICCs are not the ideal basis for reliability assessment here, 
because they tend to be low when between-group variation is low, even if there is relatively good 
agreement within groups (schools) (Jones et al, 1983). Therefore, another measure of within- 
school variation, the within-school standard error of each of these measures was also calculated, 
and their averages are shown in Table Al 1. In general, the average standard errors of the aggre- 
gated measures cover about 10-15 percent of the observed range of school averages. A confi- 
dence interval of plus and minus two standard errors around the mean would on the average 
cover 40-60 percent of the observed ranges. This means that the school-level averages are likely 
to be subject to a substantial degree of sampling error, which would likely attenuate the observed 
relationships among the school-level averages of the motivational constructs. 
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Table A13, Items and Factor Loadings from Factor Analysis of Kentucky and 
Maryland Principal Survey Responses 



Financial Incentives (Factor 1) 

Qvii6* Tripling the size of the reward available would greatly increase the motivation of teachers to meet our 
school's goals. 

Qvii4* The possibility of a bonus helps me motivate teachers to work toward the accountability goals. 

QviiS* Tripling the size of the reward available would greatly increase the motivation of teachers to meet our 
school’s goals. 

Qvii5* The bonus provides a shared goal that encourages school staff to work together. 

Qvii7* The possibility of the bonus motivates me to work toward achieving the accountability goals. 

QviilO* The bonus part of the accountability program should be continued. 

Qvii3* Is it fair for teachers who increase student achievement to receive a bonus? 

QviiiS* To what extent have you as a principal referred to the possibility of getting a bonus in discussing the 
school’s accountability goals with the teachers? 

Improvement of Teaching (Factor 2) 

To what extent: 

Qii5 have teachers worked together to plan and implement changes in what is taught or how teaching is done? 

Qii3 have teachers changed the way they teach (e.g., used different teaching methods)? 

Qiil have teachers spent more hours on teaching and teaching-related tasks? 

Qii4 have teachers engaged in professional development activities (e.g., in-service or other training) aimed at 

increasing their ability to help students achieve higher [accountability program] scores? 

Qii2 have teachers changed the content of what they teach? 

Qii9* does the typical teacher in this school continually strive to keep up with the latest developments in 
curriculum and instructional practice? 

Qii6 have teachers focused their teaching more on those aspects of student learning related to the [accountability 
program] assessments? 

Qiii20 are teachers provided time to plan together about how to improve students scores on the [accountability 
program] tests? 

District Support (Factor 3) 

To what extent: 

Qv2 do district staff support your efforts to achieve your school’s accountability goals? 

Qv3 has your district provided you with technical assistance toward achieving your school’s accountability 
goals? 

Qv5 has your district provided you with the resources you need to achieve your school’s accountability goals? 

Qvl * are district administrative staff committed to the accountability program? 

Qv6 has your district provided you the discretion to do what needs to be done to achieve your school’s 

accountability goals? 

Principal Understanding/Commitment (Factor 4) 

Qi2 I know how much each factor of the accountability index needs to improve to reach this cycle’s 
accountability goals. 

Qi4 I could explain my school’s accountability goals to others if asked. 

Qil I have a clear understanding of what my school’s accountability goals are. 

Qi3 I have a good understanding of how student achievement is measured by the . . .tests used in the 

accountability program. 

Qi9 I am strongly committed to pursuing my school’s accountability goals. 

Resource Allocation (Factor 5) 

To what extent: 

Qiiil 1 have budgeted funds been reallocated to acquire instructional materials that would contribute to better 
[accountability program] scores? 

Qiiil4 has your school invested in new technology (e.g., computers, Internet) to achieve the accountability goals? 
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Qiii 1 2 have tutoring or similar programs been established in subjects covered by the [accountability program’s] 
tests? 

Qiii 10 has staff been reallocated to assign the better teachers to the [accountability program’s] assessment grades? 
Qiii9 are incentives or rewards provided for students to do well on the tests? 

Qiii? are new curriculum development efforts linked to the [accountability program’s] assessments? 

Principal Skill (Factor 6) 

To what extent: 

Qiii 1 do you yourself have the knowledge and skills to help teachers improve their teaching? 

Qiii 2 do you have the skills needed to mobilize your school to achieve its accountability goals? 

Qiii 1 have you as a principal helped teachers figure out what they need to do to help achieve the accountability 
goals? 

QiilO do you know how to use the results of the [accountability program’s] assessments to figure out what needs 
to be done to meet your school’s accountability goals? 

Qiii36 has your school mobilized all its resources to meet its accountability goals? 

Teacher Teamwork and Learning (Factor 7) 

To what extent: 

Qiii 1 8 are teachers provided time to plan together about how to achieve the accountability goals? 

Qiii2 1 are teachers given opportunities to learn about best practices of other schools that can be used to raise 
[accountability program] scores? 

Qiii 1 9 are professional development programs focused on helping teachers acquire knowledge and skills needed to 
improve students scores on the [accountability program’s] tests? 

Qiii 1 7 are teachers in related subjects or functions encouraged to work together in teams? 

Qiii 1 5 has more emphasis been put on using assessment results to guide teaching efforts? 

* Questions not included in Charlotte- Mecklenburg Principals * Survey. 



Table A14, Coefficient Alpha Reliabilities for Factor-Based Scales 



Scale 


Alpha 


Financial Incentives 


.90 


Improvement of Teaching 


.81 


Principal Understanding 


.86 


District Support 


.89 


Resource Allocation 


.76 


Principal Skill 


.67 


Teamwork and Learning 


.77 
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Table A15. Rotated Factor Loadings Obtained from Factor Analysis of Items on the Kentucky and 

Maryland Principals’ Survey* 



Item 




1 

Finan 

Inc 


Factor 

2 

Imp 

Tea 


3 

Prin 

Und 


4 

Dis 

Supp 


5 

Res 


6 

Prin 

Skill 


7 

Team 


Qvii6 


Tripling reward motivates teachers 


.85 


.02 


.00 


.04 


-.03 


.06 


-.01 


Qvii4 


Possibility bonus helps me motivate .83 


.01 


.07 


.02 


.02 


.05 


.00 


Qvii8 


Tripling reward motivates me 


.82 


.05 


-.08 


-.01 


.03 


-.01 


.01 


QviiS 


Bonus provides shared goal 


.81 


.01 


.11 


.05 


.06 


.02 


.03 


Qvii7 


Bonus motivates me more 


.77 


.03 


-.11 


-.06 


.05 


.03 


-.06 


QviilO 


Bonus award should continue 


.76 


-.05 


.06 


.04 


-.12 


.09 


.03 


QviiS 


Fair for teachers to get reward 


.71 


-.04 


.08 


.05 


-.19 


.05 


.09 


QiiiS 


Principal referred to bonus 


.60 


.02 


-.03 


.01 


.05 


.05 


.08 


Qil5 


Teachers worked change teach. 


.03 


.75 


.04 


.08 


.05 


.20 


.24 


QiiS 


Teachers changed way teach 


.07 


.71 


.06 


.06 


-.07 


-.02 


.11 


Qiil 


Teachers spent hours teaching 


.00 


.64 


.04 


-.01 


.14 


.10 


.05 


QN4 


Teachers prof. dev. activities 


-.04 


.63 


.08 


.18 


.22 


-.01 


.06 


QN2 


Teachers changed content 


-.09 


.56 


.07 


-.08 


.28 


.02 


-.04 


Qii9 


Have knowledge help teachers 


.16 


.53 


-.01 


.11 


.04 


.26 


.25 


QN6 


Teachers focused stud, learning 


-.11 


.52 


.24 


.11 


.16 


.07 


.19 


Qiii20 


Teachers provided time plan 


.05 


.49 


.05 


.25 


.04 


.16 


.47 


Qi2 


Know improve to meet goals 


00 


.14 


.82 


.09 


.03 


.12 


.00 


Qi4 


Could explain goals to others 


-.09 


.08 


.81 


.07 


.13 


.14 


.01 


Qil 


Clear understanding goals 


.03 


-.02 


.80 


.19 


.02 


.05 


.05 


QiS 


Understanding stud. ach. meas. 


.04 


.00 


.77 


.16 


.04 


.19 


.04 


Qi9 


Committed to pursuing goals 


.01 


.22 


.50 


.14 


-.05 


.01 


.02 


Qv2 


District staff support efforts 


.02 


.08 


.12 


.84 


-.04 


.11 


.12 


Qv3 


District provided tech, assistance 


.02 


.04 


.21 


.79 


.16 


.09 


.12 


Qv5 


District provided resources 


.01 


.04 


.16 


.75 


.21 


.02 


.17 


Qvl 


District admin, staff committed 


,09 


.12 


.11 


.73 


-.13 


.16 


.01 


Qv6 


District provided you discretion 


,02 


.16 


.16 


.67 


.03 


.06 


.08 


Qiiill 


Budgeted funds reallocated 


.00 


.15 


.01 


-.01 


.73 


.08 


.16 


Qiii14 


School invested new technology 


-.01 


.03 


.05 


.12 


.72 


.06 


.23 


Qiii12 


Tutoring/programs established 


-.03 


.19 


.00 


.08 


.67 


.10 


.00 


QiiilO 


Staff realloc, assign teachers 


.00 


.21 


.13 


-.02 


.62 


.03 


-.16 


Qiii9 


Incentives/rewards for students 


.20 


.00 


.13 


.10 


.42 


.03 


.00 


Qiii7 


Newcurr. development efforts 


-.13 


.21 


.07 


.15 


.34 


.05 


.27 


Qiill 


Knowledge help teachers 


.05 


.07 


.14 


.09 


.12 


.79 


.08 


Qii12 


Skills mobilize school 


.14 


.07 


.18 


.14 


.08 


.74 


.15 


Qiiil 


Helped teachers figure out 


,09 


.22 


.22 


.08 


.00 


.59 


.22 


QiilO 


Know use assessments 


.07 


,17 


.45 


.12 


.04 


.57 


-.01 


Qiii2 


Worked individually teachers 


.09 


.30 


.19 


.10 


.05 


.48 


.20 


Qiii36 


School mobilized resources 


.09 


.29 


.20 


.23 


.12 


.31 


.20 


QiiilS 


Teachers provided time plan 


,04 


.10 


-.04 


.07 


.01 


.11 


.73 


Qiii21 


Teachers opport. best practices 


.12 


.20 


.10 


.15 


.21 


.10 


.67 


Qiii19 


Prof. dev. programs help teachers 


-.01 


.24 


.11 


.14 


.30 


.11 


.56 


Qiiil 7 


Teachers encouraged team 


,08 


.37 


-.08 


.10 


.00 


.16 


.51 


QiiilS 


Using results guide teaching 


-.03 


.31 


.12 


.18 


.28 


.02 


.34 



Varimca rotation 
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